Optical neural network system and method and apparatus for training the same

Abstract

The present disclosure provides an optical neural network system and a training method and device thereof. The method can include: performing an initial optical configuration on a main optical modulation element, i.e., initializing the phase distribution of each pixel block modulation unit to a basic phase, the phase distribution being used to determine the optical projection position of the corresponding pixel block modulation unit, the main optical modulation element including a permutation block, and the permutation block including the pixel block modulation unit; determining the initial optical configuration as a current optical configuration, and performing the following first processing: obtaining a modulated light field obtained by modulating an input light field according to the current optical configuration by the main optical modulation element, updating the weight parameters of a neural network inference unit according to the modulated light field, adjusting the phase distribution of the pixel block modulation unit in at least one permutation block if it is determined that the neural network inference unit has converged but does not meet the end condition, obtaining an updated current optical configuration, and repeating the first processing until convergence and meeting the end condition.

Description

Technical Field

[0001] This disclosure relates to fields such as optical information processing, computational imaging, and artificial intelligence, and in particular to optical neural network systems and their training methods and apparatus. Background Technology

[0002] Current edge vision systems typically employ convolutional neural networks (CNNs) for object recognition, pose detection, or state determination. The convolution operators in CNNs have inherent limitations on their local receptive fields; each kernel can only perceive information within its neighborhood. This makes it difficult for the network to directly capture long-range dependencies and global contextual features in an image. Summary of the Invention

[0003] This disclosure provides an optical neural network system and its training method and apparatus.

[0004] A training method for an optical neural network system includes:

[0005] An initial optical configuration is performed on the main optical modulation element. The optical neural network system includes the main optical modulation element and a neural network inference unit. The main optical modulation element includes at least one permutation block, and each permutation block includes at least two pixel block modulation units. The initial optical configuration includes initializing the phase distribution of each pixel block modulation unit to a predetermined base phase. The phase distribution is used to determine the optical projection position of the corresponding pixel block modulation unit. The initial optical configuration is determined as the current optical configuration, and the following first process is performed: obtaining the modulated light field obtained by the main optical modulation element modulating the input light field according to the current optical configuration; updating the weight parameters of the neural network inference unit according to the modulated light field; in response to determining that the neural network inference unit has converged but does not meet the termination condition, adjusting the phase distribution of the pixel block modulation unit in at least one of the permutation blocks so that the optical projection positions corresponding to the pixel block modulation units are exchanged to obtain the updated current optical configuration, and repeating the first process; in response to determining that the neural network inference unit has converged and meets the termination condition, ending the training.

[0006] An optical neural network system includes: a main optical modulation element and a neural network inference unit; The main optical modulation element is used to modulate the input light field according to its own optical configuration to obtain a modulated light field; The neural network inference unit is used to generate the task processing result corresponding to the modulated light field. The optical configuration of the main optical modulation element and the weight parameters of the neural network inference unit are obtained through training by the above method.

[0007] A training device for an optical neural network system includes: an initial configuration module and a collaborative training module; The initial configuration module is used to perform initial optical configuration on the main optical modulation element. The optical neural network system includes the main optical modulation element and a neural network inference unit. The main optical modulation element includes at least one permutation block, and each permutation block includes at least two pixel block modulation units. The initial optical configuration includes: initializing the phase distribution of each pixel block modulation unit to a predetermined base phase. The phase distribution is used to determine the optical projection position of the corresponding pixel block modulation unit. The collaborative training module is configured to determine the initial optical configuration as the current optical configuration and perform the following first process: obtain the modulated light field obtained by the main optical modulation element modulating the input light field according to the current optical configuration; update the weight parameters of the neural network inference unit according to the modulated light field; in response to determining that the neural network inference unit has converged but does not meet the termination condition, adjust the phase distribution of the pixel block modulation unit in at least one of the permutation blocks so that the optical projection positions corresponding to the pixel block modulation units are exchanged to obtain the updated current optical configuration, and repeat the first process; in response to determining that the neural network inference unit has converged and meets the termination condition, end the training.

[0008] An electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0009] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating an embodiment of the training method for the optical neural network system described in this disclosure; Figure 2This is a schematic diagram of the composition and structure of the main optical modulation element described in this disclosure; Figure 3 This is a schematic diagram of the composition structure of the pixel block modulation unit described in this disclosure; Figure 4 This is a schematic diagram of the horizontal flipping and replacement method described in this disclosure; Figure 5 This is a schematic diagram of the vertical flipping and replacement method described in this disclosure; Figure 6 This is a schematic diagram of the diagonal flipping permutation method described in this disclosure; Figure 7 This is a schematic diagram of the anti-angle flipping replacement method described in this disclosure; Figure 8 This is a schematic diagram of the random permutation method described in this disclosure; Figure 9 For the preset size block described in this disclosure (e.g., 2) 2) Schematic diagram of internal random permutation method; Figure 10 This is a schematic diagram of the adjacent permutation block permutation method described in this disclosure; Figure 11 This is a flowchart of an embodiment of the iterative training method described in this disclosure; Figure 12 This is a schematic diagram of the composition structure of the first embodiment of the optical neural network system described in this disclosure; Figure 13 This is a schematic diagram of the composition structure of the second embodiment of the optical neural network system described in this disclosure; Figure 14 This is a schematic diagram of the composition structure of the third embodiment of the optical neural network system described in this disclosure; Figure 15 This is a schematic diagram of the composition structure of an embodiment of the training device for the optical neural network system described in this disclosure; Figure 16 A schematic block diagram of an electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0014] Figure 1 This is a flowchart illustrating an embodiment of the training method for the optical neural network system described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0015] In step 101, an initial optical configuration is performed on the main optical modulation element. The optical neural network system includes the main optical modulation element and a neural network inference unit. The main optical modulation element includes at least one permutation block, and each permutation block includes at least two pixel patch modulation units. The initial optical configuration includes initializing the phase distribution of each pixel patch modulation unit to a predetermined base phase. The phase distribution is used to determine the optical projection position of the corresponding pixel patch modulation unit.

[0016] In step 102, the initial optical configuration is determined as the current optical configuration, and the following first process is performed: obtaining the modulated light field obtained by the main optical modulation element modulating the input light field according to the current optical configuration; updating the weight parameters of the neural network inference unit according to the modulated light field; in response to determining that the neural network inference unit has converged but does not meet the termination condition, adjusting the phase distribution of the pixel block modulation unit in at least one permutation block so that the optical projection position corresponding to the pixel block modulation unit is exchanged to obtain the updated current optical configuration, and repeating the first process; in response to determining that the neural network inference unit has converged and meets the termination condition, ending the training.

[0017] The scheme described in the above-described method embodiment utilizes a master optical modulation element to pre-modulate the input light field in the optical domain. By adjusting the phase distribution of pixel block modulation units in at least one permutation block, the optical projection positions corresponding to different pixel block modulation units are exchanged. This reorganizes and converges information from different spatial regions in the input light field, allowing pixel information originally dispersed over a large spatial area to be mapped to a local neighborhood. Consequently, the neural network inference unit can obtain a larger effective receptive field and stronger information mixing capability with a smaller convolution kernel, improving the accuracy of task processing results without increasing network depth or computational overhead.

[0018] In addition, in some embodiments of this disclosure, updating the weight parameters of the neural network inference unit according to the modulated light field, and adjusting the phase distribution of the pixel block modulation unit in at least one permutation block in response to determining that the neural network inference unit has converged but does not meet the termination condition, may include performing the following second process: obtaining the task processing result corresponding to the modulated light field generated by the neural network inference unit, determining the loss function value according to the task processing result, updating the weight parameters of the neural network inference unit according to the loss function value in response to determining that the neural network inference unit has not converged, and repeating the second process, and adjusting the phase distribution of the pixel block modulation unit in at least one permutation block in response to determining that the neural network inference unit has converged but does not meet the termination condition.

[0019] Specifically, an initial optical configuration is first performed on the main optical modulation element. In some embodiments of this disclosure, the main optical modulation element may include at least one permutation block, and each permutation block may include at least two pixel block modulation units. Accordingly, the initial optical configuration may include: initializing the phase distribution of each pixel block modulation unit to a predetermined base phase. Further, the initial optical configuration may be determined as the current optical configuration for subsequent iterative training. In addition, updating the current optical configuration may include: updating the pixel block permutation rules of at least one permutation block, adjusting the phase distribution of the corresponding pixel block modulation units according to the updated pixel block permutation rules, to obtain the updated current optical configuration.

[0020] This embodiment further optimizes the optical configuration of the main optical modulation element and the weight parameters of the neural network inference unit through alternating iterations and collaborative optimization, enabling them to adapt to each other and converge together during training. On one hand, after the neural network inference unit converges under the current optical configuration, adjusting the phase distribution of the pixel block modulation units to exchange the optical projection positions corresponding to different pixel block modulation units provides the neural network inference unit with a new light field distribution after information rearrangement, breaking the limitation of its local receptive field. On the other hand, the updated light field distribution provides the neural network inference unit with a new learning target, driving its weight parameters to continue to optimize. Through this alternating process, the optimal optical configuration and weight parameters for the overall system performance are ultimately determined.

[0021] Preferably, the main optical modulation element can be a metasurface used to modulate the input light field to obtain a modulated light field, that is, to rearrange the input light field at the pixel block level.

[0022] Figure 2 This is a schematic diagram of the composition and structure of the main optical modulation element described in this disclosure. Figure 2 As shown, the main optical modulation element can be divided into at least one permutation block, typically multiple, the specific number depending on actual needs. Each permutation block can be composed of M... It consists of N pixel blocks arranged in an array, where M and N can be equal or unequal, and their specific values ​​can be determined according to actual needs. For example Figure 2 Both M and N shown can be 3. A base phase can be set for each pixel block modulation unit. Accordingly, the initial optical configuration refers to initializing the phase distribution of each pixel block modulation unit to its corresponding base phase.

[0023] In some embodiments of this disclosure, the optical neural network system may further include: a photosensitive element for converting a modulated light field into a modulated signal via photoelectric conversion. Each pixel block modulation unit may correspond to a pixel block in the photosensitive element, and its size matches that of the corresponding pixel block.

[0024] The photosensitive element can be a complementary metal-oxide-semiconductor (CMOS) or a charge-coupled device (CCD), etc., and is set in the output optical path of the main optical modulation element to receive the modulated light field and convert it into a modulated signal in the form of an electrical signal.

[0025] Each pixel block modulation unit can contain multiple sub-units, and the relative spatial relationship between the sub-units remains unchanged, thus maintaining the structural integrity within the pixel block modulation unit.

[0026] Figure 3 This is a schematic diagram illustrating the structural composition of the pixel block modulation unit described in this disclosure. Figure 3 As shown, each pixel block modulation unit can contain n n (as shown in the figure) Sub-units are arranged in an array. The number of sub-units is the same as the number of pixels in the corresponding pixel block. Each sub-unit is a subwavelength-scale structural unit (such as nanorods, nanopillars, etc.), and its geometric parameters (size, orientation, shape) are designed to give it a specific phase response.

[0027] During operation, the sub-units within each pixel block modulation unit work together to project the light from the input light field region corresponding to the pixel block modulation unit onto the photosensitive element at the pixel block position corresponding to the current position of the pixel block modulation unit.

[0028] The pixel block replacement rule uses pixel block modulation units as the basic unit. By adjusting the phase distribution of each pixel block modulation unit (superimposing the phase shift required for replacement on top of the basic phase), the optical projection positions between pixel block modulation units are exchanged. Each pixel block modulation unit corresponds to a different optical projection position. Updating the pixel block replacement rule means replacing the projection target corresponding to the pixel block modulation unit, so that the modulated light is guided to the replaced pixel block position. In other words, the actual spatial position of the pixel block modulation unit on the main optical modulation element does not change; what changes is the pixel block to which the light it controls ultimately strikes.

[0029] The mathematical model for the above position exchange can be expressed as: y=Pgx; (1) in, This represents the vectorized representation of the input light field. The vectorized representation of the output light field (modulated light field) is given by Pg, where Pg represents the permutation matrix and g represents the pixel block permutation rule (i.e., the exchange mapping relationship between each pixel block modulation unit).

[0030] The phase distribution of the corresponding pixel block modulation unit is adjusted according to the updated pixel block replacement rule to obtain the updated current optical configuration. This means reconfiguring the main optical modulation element according to the updated pixel block replacement rule. For example, by adjusting the size and orientation of the nanopillars of each sub-unit or changing the refractive index of the material by applying an external voltage, the phase distribution of the corresponding pixel block modulation unit is changed, so that the optical response of the main optical modulation element accurately matches the updated pixel block replacement rule. The input light field is then rearranged according to the updated pixel block replacement rule and output to the photosensitive element.

[0031] Multiple pixel block permutation rules can be predefined in the pixel block permutation rule library, and the specific rules included can be determined according to actual needs. For example, it can include horizontal flip permutation, vertical flip permutation, main diagonal flip permutation, anti-diagonal flip permutation, random permutation, random permutation within a preset size block, and permutation of adjacent permutation blocks, etc. In addition, it can also include some more complex pixel block permutation rules, such as identity mapping that maintains the original spatial structure, convergence permutation that gathers scattered pixel blocks into a local area, scattered permutation, hybrid permutation, periodic shuffling, etc., thus providing a rich set of rule options for optical modulation to adapt to different task requirements.

[0032] Figure 4 This is a schematic diagram of the horizontal flipping and replacement method described in this disclosure. Figure 5 This is a schematic diagram of the vertical flipping and replacement method described in this disclosure. Figure 6 This is a schematic diagram of the diagonal flipping permutation method described in this disclosure. Figure 7 This is a schematic diagram of the anti-angle flipping permutation method described in this disclosure. Figure 8This is a schematic diagram of the random permutation method described in this disclosure. Figure 9 For the preset size block described in this disclosure (e.g., 2) 2) Schematic diagram of internal random permutation method, Figure 10 This is a schematic diagram of the adjacent permutation block permutation method described in this disclosure.

[0033] In some embodiments of this disclosure, the optical neural network system may further include: an auxiliary optical modulation element for preprocessing the input light field and then transmitting it to the main optical modulation element.

[0034] The auxiliary optical modulation element can be a conventional optical element used to preprocess the input light field, such as focusing and collimating, to improve the quality of the light field transmitted to the main optical modulation element. If necessary, the auxiliary optical modulation element can also be a metasurface element, and can work with the main optical modulation element to perform pixel-level rearrangement of the input light field.

[0035] Based on the above description, the overall workflow of the optical neural network system can be summarized as follows: An auxiliary optical modulation element preprocesses the input light field and transmits the preprocessed input light field to the main optical modulation element. The main optical modulation element modulates the preprocessed input light field according to the current optical configuration to obtain a modulated light field. The photosensitive element converts the modulated light field into a modulation signal through photoelectric conversion. This modulation signal is then input to the neural network inference unit, which generates the corresponding task processing result. The input light field can be the original image light field, etc.

[0036] The neural network inference unit can perform feature extraction and task inference based on the modulated signal to obtain the task processing result. It can be implemented using a convolutional neural network, a transform network, or a combination of both, and works in conjunction with the optical permutation of the main optical modulation element. The main optical modulation element completes spatial information rearrangement in the optical domain, while the neural network inference unit performs feature extraction and task inference in the digital domain, together achieving efficient and low-latency visual processing.

[0037] After initial optical configuration of the main optical modulation element, the initial optical configuration can be determined as the current optical configuration, and iterative training can be performed. For example, the following first process is executed: Obtain the modulated light field obtained after the main optical modulation element modulates the input light field according to the current optical configuration. Then, the following second process is executed: Obtain the task processing result corresponding to the modulated light field generated by the neural network inference unit; determine the loss function value based on the task processing result; in response to determining that the neural network inference unit has not converged, update the weight parameters of the neural network inference unit based on the loss function value, and repeat the second process. In response to determining that convergence has been achieved but the termination condition is not met, update the current optical configuration and repeat the first process; in response to determining that convergence has been achieved and the termination condition is met, end the training.

[0038] Accordingly, Figure 11 This is a flowchart illustrating an embodiment of the iterative training method described in this disclosure. Figure 11 As shown, the specific implementation methods are as follows.

[0039] In step 1101, the modulated light field obtained by the main optical modulation element modulating the input light field according to the current optical configuration is acquired.

[0040] The input light field here can refer to the input light field after preprocessing by auxiliary optical modulation elements.

[0041] Based on the current optical configuration, each sub-unit in the main optical modulation element can deflect light from a specific spatial position in the input light field to a designated pixel block position on the photosensitive element.

[0042] In step 1102, the modulation signal is obtained by the photosensitive element through photoelectric conversion of the modulation light field into a modulation signal.

[0043] In step 1103, the modulated signal is input into the neural network inference unit to obtain the output task processing result.

[0044] Depending on the specific task scenario, the task processing results may include: image classification results, object detection results, semantic segmentation results, face recognition results, etc.

[0045] In step 1104, the loss function value is determined according to the task processing result and the expected processing result corresponding to the input light field, in accordance with the loss function value determination method that matches the task type corresponding to the task processing result.

[0046] The expected processing results can be generated through manual annotation or other feasible methods.

[0047] The loss function value can be calculated based on the difference between the task processing result and the expected processing result. For example, the cross-entropy loss function can be used to determine the loss function value for classification tasks, the weighted sum of the bounding box regression loss and classification loss can be used for detection tasks, and the pixel block cross-entropy loss function can be used for segmentation tasks. This enables differentiated loss calculation for different task types, and thus allows for a more accurate evaluation of the task performance of the neural network inference unit under the current optical configuration, providing a quantitative basis for subsequent weight parameter updates and pixel block replacement rule optimization.

[0048] In step 1105, it is determined whether the neural network inference unit has converged. If it has not converged, step 1106 is executed. If it has converged, step 1107 is executed.

[0049] For example, the neural network inference unit can be determined to have converged when the second process has been executed T times. Alternatively, the neural network inference unit can be determined to have converged when the difference between the loss function value obtained in R consecutive iterations and the loss function value obtained in the previous iteration is less than a difference threshold. T and R are both positive integers greater than 1, and the specific values ​​of T, R, and the difference threshold can be determined according to actual needs.

[0050] In other words, convergence can be considered achieved if the maximum number of training rounds is reached (to prevent infinite loops), or convergence can be considered achieved by monitoring the magnitude of changes in the loss function value and seeing that the changes are less than the difference threshold for multiple consecutive rounds. Accordingly, it is possible to accurately determine whether the neural network inference unit has converged to the optimal state under the current optical configuration, avoiding under-optimization or excessive iteration, and improving the stability and efficiency of the collaborative optimization process.

[0051] In step 1106, the weight parameters of the neural network inference unit are updated according to the loss function value, and then step 1103 is executed.

[0052] For example, the weight parameters of the neural network inference unit can be updated based on the loss function value using the backpropagation algorithm and gradient descent optimizer, and then the process returns to step 1103. That is, the modulated signal is reprocessed based on the updated weight parameters, and the inner loop optimization of the neural network inference unit continues.

[0053] In step 1107, determine whether the termination condition is met. If yes, the process ends; otherwise, proceed to step 1108 to start a new round of collaborative optimization loop.

[0054] For example, determining that the termination condition is met can mean determining that the first condition or the second condition is met. The first condition can be that the loss function value is less than or equal to the loss threshold, and the second condition can be that the first process is executed Q times, where Q is a positive integer greater than 1.

[0055] The specific values ​​of Q and the loss threshold can be determined according to actual needs.

[0056] In other words, if the maximum number of training epochs is reached (to prevent infinite loops), the termination condition can be considered met. Alternatively, the loss function value can be compared with a pre-set loss threshold; if the loss function value is less than or equal to the loss threshold, the termination condition can also be considered met. This effectively avoids infinite loops in the optimization process, accurately determines whether the system has reached the preset performance target, ensures stable convergence of co-optimization within a controllable period, and improves the robustness and practicality of the overall solution.

[0057] In step 1108, the pixel block permutation rule of at least one permutation block is updated, and the phase distribution of the corresponding pixel block modulation unit is adjusted according to the updated pixel block permutation rule to obtain the updated current optical configuration. Then, step 1101 is executed.

[0058] In practical applications, the pixel block replacement rules can be updated for only one or more replacement blocks, or for all replacement blocks. The specific method is not limited and can be determined according to actual needs.

[0059] Additionally, updating the pixel block replacement rule for at least one permutation block can include: determining the updated pixel block replacement rule for each permutation block by using the same pixel block replacement rule for all permutation blocks; or, determining the updated pixel block replacement rule for each permutation block separately based on the location of each permutation block. The former is a globally consistent mode, where all permutation blocks use the same pixel block replacement rule to achieve a uniform global permutation effect. The latter is a regionally differentiated mode, where different permutation blocks can use different pixel block replacement rules based on their spatial location, such as using an identity mapping that preserves local structure in the image center region, and using convergent permutation in edge regions to introduce remote contextual information, etc.

[0060] Furthermore, the updated pixel block replacement rules may include: unused pixel block replacement rules randomly selected from the available pixel block replacement rules (which may refer to all pixel block replacement rules in the pixel block replacement rule library, or a selected portion of pixel block replacement rules). Alternatively, unused pixel block replacement rules selected from the available pixel block replacement rules based on the task type corresponding to the task processing result or the light field characteristics of the input light field.

[0061] For example, for a specific task type, the priorities of different available pixel block replacement rules can be pre-set, and they can be tried sequentially during updates. That is, if the termination condition is not met under the pixel block replacement rule of the current priority, the system switches to the pixel block replacement rule of the next higher priority. Alternatively, for a specific task type, during updates, unselected pixel block replacement rules can be randomly selected from the available different pixel block replacement rules for trying, without priority distinction, to achieve a more comprehensive rule exploration.

[0062] For example, for the input light field, specified features can be extracted, and updates can be attempted sequentially according to the priority of different pixel block replacement rules corresponding to the specified features. The specified features can refer to a single feature or a combination of multiple features, such as "high texture + random", "strong edge + large target", "weak edge + low texture", "medium texture + periodicity", "multi-scale + complexity", etc.

[0063] Through the above processing, a diverse and highly adaptable selection method can be provided for pixel block replacement rules, thereby effectively expanding the rule exploration space, improving the matching degree of optical modulation with different visual tasks and different light field characteristics, and enhancing system adaptability and optimization effect.

[0064] After updating the pixel block permutation rules of at least one permutation block, the phase distribution of the corresponding pixel block modulation units can be adjusted according to the updated pixel block permutation rules to obtain the updated current optical configuration. This allows for the reconfiguration of the main optical modulation element based on the updated pixel block permutation rules. Specifically, the pixel block permutation rules operate at the pixel block modulation unit level, adjusting the phase distribution of the corresponding pixel block modulation units (by superimposing the phase shift required for permutation onto the base phase) to achieve position exchange. For example, light originally controlled by pixel block modulation unit a is now controlled by pixel block modulation unit b, while light originally controlled by pixel block modulation unit b is now controlled by pixel block modulation unit a. This exchange causes a corresponding permutation of the pixel block positions on the photosensitive element corresponding to the light from each pixel block modulation unit, thereby completing the pixel-level spatial information rearrangement in the optical domain.

[0065] In some embodiments of this disclosure, after each determination of the loss function value, the loss function value can be compared with the historical best loss value. In response to determining that the loss function value is less than the historical best loss value, the historical best loss value can be updated to the loss function value, and the current optical configuration and the current weight parameters of the neural network inference unit can be determined as the optimal configuration corresponding to the historical best loss value. In response to meeting a first condition, the current optical configuration and the weight parameters of the neural network inference unit at the end of training can be determined as the target training result. In response to determining that meeting a second condition, the optimal configuration corresponding to the latest obtained historical best loss value can be determined as the target training result.

[0066] It can be executed Figure 11 During the illustrated process, the optimal configuration is continuously recorded. Each time a loss function value is determined, it is compared with the historical best loss value. If the loss function value is less than the historical best loss value, the historical best loss value is updated using the new loss function value, and the current optical configuration and the current weight parameters of the neural network inference units are saved as the optimal configuration. Thus, upon termination of training, if the first condition is met (i.e., the process ends normally), the latest training result is the desired target training result. If training is terminated due to reaching the maximum number of training iterations (i.e., the process is forcibly terminated), the optimal configuration recorded during training (i.e., the current optical configuration and weight parameters corresponding to the minimum loss function value) can be used as the desired target training result.

[0067] Accordingly, through the above processing, the globally optimal training result can be output regardless of whether the training process ends normally or is forcibly terminated, thereby significantly improving the stability of system training and final inference performance, avoiding the risk of losing the optimal configuration due to forced truncation, and improving the consistency and reliability of collaborative optimization effects.

[0068] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0069] The above is an introduction to the method embodiments. The following system and device embodiments will further illustrate the solution described in this disclosure.

[0070] Figure 12 This is a schematic diagram of the structural composition of the first embodiment 1200 of the optical neural network system described in this disclosure. Figure 12 As shown, it includes: a main optical modulation element 1201 and a neural network inference unit 1202.

[0071] The main optical modulation element 1201 is used to modulate the input light field according to its own optical configuration to obtain a modulated light field.

[0072] The neural network inference unit 1202 is used to generate the task processing result corresponding to the modulated light field. The optical configuration of the main optical modulation element 1201 and the weight parameters of the neural network inference unit 1202 are obtained by training through the aforementioned method.

[0073] Figure 13 This is a schematic diagram of the structural composition of the second embodiment 1300 of the optical neural network system described in this disclosure. Figure 13 As shown, it includes: a main optical modulation element 1201, a neural network inference unit 1202, and a photosensitive element 1203.

[0074] It can be seen that, compared to Figure 12 The optical neural network system shown Figure 13 The optical neural network system shown further includes a photosensitive element 1203 for converting the modulated light field into a modulated signal through photoelectric conversion.

[0075] Additionally, the main optical modulation element 1201 may include at least one permutation block, and each permutation block may include at least two pixel block modulation units. Each pixel block modulation unit may correspond to a pixel block in the photosensitive element, and its size matches the corresponding pixel block. Furthermore, the neural network inference unit 1202 can generate task processing results based on the input modulation signal.

[0076] Figure 14This is a schematic diagram of the structural composition of the third embodiment 1400 of the optical neural network system described in this disclosure. Figure 14 As shown, it includes: a main optical modulation element 1201, a neural network inference unit 1202, a photosensitive element 1203, and an auxiliary optical modulation element 1204.

[0077] It can be seen that, compared to Figure 13 The optical neural network system shown, Figure 14 The illustrated optical neural network system further includes an auxiliary optical modulation element 1204 for preprocessing the input light field before transmitting it to the main optical modulation element 1201. The preprocessing may include focusing and collimation, etc.

[0078] In the above embodiments, the main optical modulation element 1201 may be a metasurface element.

[0079] Figure 15 This is a schematic diagram of the structural composition of an embodiment 1500 of the training device for the optical neural network system described in this disclosure. Figure 15 As shown, it includes: an initial configuration module 1501 and a collaborative training module 1502.

[0080] The initial configuration module 1501 is used to perform initial optical configuration on the main optical modulation element. The optical neural network system includes the main optical modulation element and a neural network inference unit. The main optical modulation element includes at least one permutation block, and each permutation block includes at least two pixel block modulation units. The initial optical configuration includes: initializing the phase distribution of each pixel block modulation unit to a predetermined base phase, wherein the phase distribution is used to determine the optical projection position of the corresponding pixel block modulation unit.

[0081] The collaborative training module 1502 is used to determine the initial optical configuration as the current optical configuration and perform the following first process: obtain the modulated light field obtained by the main optical modulation element modulating the input light field according to the current optical configuration; update the weight parameters of the neural network inference unit according to the modulated light field; in response to determining that the neural network inference unit has converged but does not meet the termination condition, adjust the phase distribution of the pixel block modulation unit in at least one permutation block so that the optical projection position corresponding to the pixel block modulation unit is exchanged to obtain the updated current optical configuration, and repeat the first process; in response to determining that the neural network inference unit has converged and meets the termination condition, end the training.

[0082] In some embodiments of this disclosure, the collaborative training module 1502 updates the weight parameters of the neural network inference unit according to the modulated light field. In response to determining that the neural network inference unit has converged but does not meet the termination condition, the method of adjusting the phase distribution of the pixel block modulation unit in at least one permutation block may include: performing the following second process: obtaining the task processing result corresponding to the modulated light field generated by the neural network inference unit, determining the loss function value according to the task processing result, in response to determining that the neural network inference unit has not converged, updating the weight parameters of the neural network inference unit according to the loss function value, and repeating the second process, in response to determining that the neural network inference unit has converged but does not meet the termination condition, adjusting the phase distribution of the pixel block modulation unit in at least one permutation block.

[0083] In addition, in some embodiments of this disclosure, the collaborative training module 1502 adjusts the phase distribution of the pixel block modulation units in at least one of the permutation blocks to obtain an updated current optical configuration by: updating the pixel block permutation rules of at least one permutation block, adjusting the phase distribution of the corresponding pixel block modulation units according to the updated pixel block permutation rules, and obtaining an updated current optical configuration.

[0084] In some embodiments of this disclosure, the optical neural network system may further include: a photosensitive element for converting a modulated light field into a modulated signal through photoelectric conversion; each pixel block modulation unit may correspond to a pixel block in the photosensitive element, and the size of the unit may match the size of the corresponding pixel block. Additionally, the collaborative training module 1502 may input the modulated signal into the neural network inference unit to obtain the output task processing result.

[0085] In some embodiments of this disclosure, the optical neural network system may further include: an auxiliary optical modulation element for preprocessing the input light field and then transmitting it to the main optical modulation element.

[0086] In some embodiments of this disclosure, the collaborative training module 1502 may determine the loss function value based on the task processing result in a manner that matches the task type corresponding to the task processing result, and determine the loss function value based on the task processing result and the expected processing result corresponding to the input light field.

[0087] In some embodiments of this disclosure, the collaborative training module 1502 may update the pixel block permutation rules of at least one permutation block by: determining the updated pixel block permutation rules for each permutation block according to the same pixel block permutation rules for each permutation block; or, determining the updated pixel block permutation rules for each permutation block separately according to the different positions of each permutation block.

[0088] In some embodiments of this disclosure, the updated pixel block replacement rule may include: an unused pixel block replacement rule randomly selected from the available pixel block replacement rules. Alternatively, an unused pixel block replacement rule may be selected from the available pixel block replacement rules based on the task type corresponding to the task processing result or the light field characteristics of the input light field.

[0089] In some embodiments of this disclosure, the collaborative training module 1502 determines that the neural network inference unit has converged in the following ways: It determines that the neural network inference unit has converged in response to determining that the second processing has been executed T times. Alternatively, it determines that the neural network inference unit has converged in response to determining that the difference between the loss function value obtained in R consecutive iterations and the loss function value obtained in the previous iteration is less than a difference threshold, where T and R are both positive integers greater than 1. Determining that the termination condition is met may include: determining that a first condition or a second condition is met, where the first condition is that the loss function value is less than or equal to the loss threshold, and the second condition is that the first processing has been executed Q times, where Q is a positive integer greater than 1.

[0090] In some embodiments of this disclosure, after the collaborative training module 1502 determines the loss function value each time, it can also compare the loss function value with the historical best loss value. In response to determining that the loss function value is less than the historical best loss value, the historical best loss value can be updated to the loss function value, and the current optical configuration and the current weight parameters of the neural network inference unit can be determined as the optimal configuration corresponding to the historical best loss value. Accordingly, in response to meeting the first condition, the current optical configuration and the weight parameters of the neural network inference unit at the end of training can be determined as the target training result. In response to determining that meeting the second condition, the optimal configuration corresponding to the latest obtained historical best loss value can be determined as the target training result.

[0091] The specific workflow of the above system and device embodiments can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0092] In summary, the scheme described in this disclosure can extend the receptive field, which relies on deep networks in traditional methods, to the optical domain through pixel block replacement. This allows pixel block information scattered across different spatial locations in the original image to be reorganized into the same local neighborhood. Without deepening the network structure, the effective receptive field of the neural network inference unit is effectively expanded, improving the locality limitations of traditional convolution. Specifically, by designing corresponding pixel block replacement rules, distant and scattered pixel block information in the original image can be converged into the same local convolution window. This allows spatially dispersed information to be aggregated in advance in the optical domain, achieving equivalent coverage of a large original area by a small convolution kernel, thus achieving a "small convolution kernel, large field of view" optical-digital collaborative processing effect. For example, for a K×K convolution kernel, after pixel block rearrangement, the effective receptive field can be extended to the entire replacement block range. For instance, a 3×3 convolution kernel can cover a 9×9 original image area, effectively doubling the receptive field. In addition, under the same receptive field requirements, this solution can significantly reduce the number of neural network layers, such as from 7-8 layers to 3-4 layers, which significantly reduces the computational overhead, storage pressure and system power consumption in the digital domain, making it more suitable for edge computing, edge devices and mobile deployment scenarios.

[0093] The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0094] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0095] Figure 16 A schematic block diagram of an electronic device 1600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0096] like Figure 16As shown, the electronic device 1600 includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1602 or a computer program loaded from a storage unit 1608 into a random access memory (RAM) 1603. The RAM 1603 may also store various programs and data required for the operation of the electronic device 1600. The computing unit 1601, ROM 1602, and RAM 1603 are interconnected via a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.

[0097] Multiple components in electronic device 1600 are connected to I / O interface 1605, including: input unit 1606, such as keyboard, mouse, etc.; output unit 1607, such as various types of displays, speakers, etc.; storage unit 1608, such as disk, optical disk, etc.; and communication unit 1609, such as network card, modem, wireless transceiver, etc. Communication unit 1609 allows electronic device 1600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0098] The computing unit 1601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 1601 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1600 via ROM 1602 and / or communication unit 1609. When the computer program is loaded into RAM 1603 and executed by the computing unit 1601, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 1601 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0099] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0100] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0101] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0103] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0104] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0105] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0106] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for an optical neural network system, characterized in that, include: The main optical modulation element is initially optically configured, and the optical neural network system includes the main optical modulation element and a neural network inference unit. The main optical modulation element includes at least one permutation block, and each permutation block includes at least two pixel block modulation units. The initial optical configuration includes: initializing the phase distribution of each pixel block modulation unit to a predetermined base phase, wherein the phase distribution is used to determine the optical projection position of the corresponding pixel block modulation unit. The initial optical configuration is determined as the current optical configuration, and the following first process is performed: obtaining the modulated light field obtained by the main optical modulation element modulating the input light field according to the current optical configuration; updating the weight parameters of the neural network inference unit according to the modulated light field; in response to determining that the neural network inference unit has converged but does not meet the termination condition, adjusting the phase distribution of the pixel block modulation unit in at least one of the permutation blocks so that the optical projection positions corresponding to the pixel block modulation units are exchanged, obtaining the updated current optical configuration, and repeating the first process; in response to determining that the neural network inference unit has converged and meets the termination condition, ending the training. The optical neural network system further includes: a photosensitive element for converting the modulated light field into a modulated signal through photoelectric conversion; each pixel block modulation unit corresponds to a pixel block in the photosensitive element.

2. The method according to claim 1, characterized in that, The step of updating the weight parameters of the neural network inference unit according to the modulated light field, and adjusting the phase distribution of the pixel block modulation units in at least one of the permutation blocks in response to determining that the neural network inference unit has converged but does not meet the termination condition, includes: The following second process is performed: obtaining the task processing result corresponding to the modulated light field generated by the neural network inference unit, determining the loss function value based on the task processing result, updating the weight parameters of the neural network inference unit based on the loss function value in response to determining that the neural network inference unit has not converged, and repeating the second process; adjusting the phase distribution of the pixel block modulation unit in at least one of the permutation blocks in response to determining that the neural network inference unit has converged but does not meet the termination condition.

3. The method according to claim 1, characterized in that, The step of adjusting the phase distribution of the pixel block modulation units in at least one of the displacement blocks to obtain the updated current optical configuration includes: The pixel block permutation rule of at least one of the permutation blocks is updated, and the phase distribution of the corresponding pixel block modulation unit is adjusted according to the updated pixel block permutation rule to obtain the updated current optical configuration.

4. The method according to claim 2, characterized in that, The step of obtaining the task processing result corresponding to the modulated light field generated by the neural network inference unit includes: inputting the modulated signal into the neural network inference unit to obtain the output task processing result.

5. The method according to claim 3, characterized in that, The pixel block permutation rule for updating at least one of the permutation blocks includes: The updated pixel block replacement rules for each replacement block are determined by using the same pixel block replacement rules for each replacement block. Alternatively, the pixel block replacement rules updated for each replacement block can be determined according to the different positions of each replacement block.

6. The method according to claim 2, characterized in that, The updated pixel block replacement rules include: unused pixel block replacement rules randomly selected from the available pixel block replacement rules, or unused pixel block replacement rules selected from the available pixel block replacement rules based on the task type corresponding to the task processing result or the light field characteristics of the input light field.

7. The method according to claim 2, characterized in that, Determining whether the termination condition is met includes: determining whether a first condition or a second condition is met, wherein the first condition is that the loss function value is less than or equal to the loss threshold, and the second condition is that the first process is executed Q times, where Q is a positive integer greater than 1.

8. An optical neural network system, characterized in that, include: Main optical modulation element and neural network inference unit; The main optical modulation element is used to modulate the input light field according to its own optical configuration to obtain a modulated light field; The neural network inference unit is used to generate the task processing result corresponding to the modulated light field, and the optical configuration of the main optical modulation element and the weight parameters of the neural network inference unit are obtained by training using the method described in any one of claims 1 to 7.

9. The system according to claim 8, characterized in that, The optical neural network system further includes: a photosensitive element, used to convert the modulated light field into a modulated signal through photoelectric conversion; The main optical modulation element includes at least one displacement block, and each displacement block includes at least two pixel block modulation units, each pixel block modulation unit corresponding to a pixel block of the photosensitive element. The neural network inference unit generates the task processing result based on the input modulation signal.

10. The system according to claim 8, characterized in that, The optical neural network system further includes an auxiliary optical modulation element, used to preprocess the input light field and then transmit it to the main optical modulation element.

11. The system according to claim 8, characterized in that, The main optical modulation element includes a metasurface element.

12. A training device for an optical neural network system, characterized in that, include: Initial configuration module and collaborative training module; The initial configuration module is used to perform initial optical configuration on the main optical modulation element. The optical neural network system includes the main optical modulation element and a neural network inference unit. The main optical modulation element includes at least one permutation block, and each permutation block includes at least two pixel block modulation units. The initial optical configuration includes: initializing the phase distribution of each pixel block modulation unit to a predetermined base phase. The phase distribution is used to determine the optical projection position of the corresponding pixel block modulation unit. The collaborative training module is used to determine the initial optical configuration as the current optical configuration and perform the following first process: obtain the modulated light field obtained by the main optical modulation element modulating the input light field according to the current optical configuration; update the weight parameters of the neural network inference unit according to the modulated light field; in response to determining that the neural network inference unit has converged but does not meet the termination condition, adjust the phase distribution of the pixel block modulation unit in at least one of the permutation blocks so that the optical projection positions corresponding to the pixel block modulation units are exchanged to obtain the updated current optical configuration, and repeat the first process; in response to determining that the neural network inference unit has converged and meets the termination condition, end the training. The optical neural network system further includes: a photosensitive element for converting the modulated light field into a modulated signal through photoelectric conversion; each pixel block modulation unit corresponds to a pixel block in the photosensitive element.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images