Image classification model, training method thereof and image classification method

Abstract

The invention discloses an image classification model and a training method thereof, and an image classification method, the image classification model comprises at least two metasurfaces which are sequentially arranged along a direction from an input end to an output end, the training method comprises the following steps: obtaining a training information set which comprises a plurality of groups of training samples, each group of training samples comprises a sample image and at least two sample labels in one-to-one correspondence with the metasurfaces, the sample labels represent light intensity distribution images of different preset imaging areas, and each preset imaging area corresponds to an image category; and based on a training information set, training the at least two metasurfaces in sequence along the direction from the input end to the output end of the image classification model, so that a light intensity distribution image obtained after the trained metasurfaces process the sample image and a corresponding sample label meet a preset condition. According to the method provided by the invention, the problem of gradient disappearance accumulated in the transmission process of the gradient in traditional back propagation can be avoided.

Description

Technical Field

[0001] The present invention relates to the field of optical design technology, and in particular to an image classification model and a training method thereof, and an image classification method. Background Art

[0002] Optical neural networks have become a hot topic of research and attention in this field due to their unique parallel processing capabilities, information transmission capabilities, and advantages in direct image processing.

[0003] In the training process of optical neural networks based on metasurfaces, existing methods mainly simulate the training steps of electronic neural networks through the back-propagation method, compare the results obtained on CMOS (Complementary Metal-Oxide-Semiconductor) with the expected results, calculate the loss function, and reversely update the configuration parameters on the metasurface imaging device through the loss function.

[0004] However, due to the physical properties of light, such as diffraction and scattering, the optical signal will gradually attenuate during transmission, resulting in a significant reduction in the accuracy and strength of deep network control during back propagation. This attenuation phenomenon is particularly obvious after the optical neural network exceeds five layers, which seriously restricts the application of optical neural networks in the field of deep learning. Especially in image classification, when the back propagation algorithm updates the configuration parameters of the metasurface imaging device, it is difficult to accurately capture and transmit the gradient information in the deep network. As a result, in image classification tasks, as the number of network layers increases, the classification accuracy and generalization ability are often seriously affected. Summary of the invention

[0005] One of the purposes of the present invention is to provide an image classification model training method to solve the technical problem in the prior art that gradient vanishing is prone to occur when training a neural network model through back propagation, resulting in slow model training speed.

[0006] One of the objectives of the present invention is to provide an image classification model.

[0007] One of the objectives of the present invention is to provide an image classification method.

[0008] One of the objectives of the present invention is to provide an electronic device.

[0009] One of the objects of the present invention is to provide a computer storage medium.

[0010] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides an image classification model training method, wherein the image classification model includes at least two hypersurfaces arranged in sequence along the direction from the input end to the output end, and the training method includes: obtaining a training information set, wherein the training information set includes multiple groups of training samples, each group of the training samples includes a sample image and at least two sample labels corresponding to the hypersurface one by one, wherein the sample labels represent the light intensity distribution images of different preset imaging areas, and each of the preset imaging areas corresponds to an image category; based on the training information set, the at least two hypersurfaces are trained in sequence along the direction from the input end to the output end of the image classification model, so that the light intensity distribution image obtained after the trained hypersurface processes the sample image and the corresponding sample label meet preset conditions.

[0011] As a further improvement of an embodiment of the present invention, based on the training information set, the at least two hypersurfaces are trained in sequence along the input end to the output end of the image classification model, including: based on the training information set, training the first hypersurface to be trained, the first hypersurface to be trained is the first hypersurface close to the input end of the image classification model; and / or, based on the first light intensity distribution image output by the previous hypersurface of the second hypersurface to be trained, training the second hypersurface to be trained is a hypersurface other than the first hypersurface to be trained.

[0012] As a further improvement of an embodiment of the present invention, the method also includes: based on the first light intensity distribution image corresponding to the training sample or the current hypersurface, using the current hypersurface to process and obtain a second light intensity distribution image; calculating the loss value of the second light intensity distribution image based on the loss function; and updating the configuration parameters of the current hypersurface based on the loss value until the preset conditions are met.

[0013] As a further improvement of an embodiment of the present invention, the calculating the loss value of the second light intensity distribution image based on the loss function includes: calculating the loss value of the second light intensity distribution image based on the following loss function: G = ∑relu (h i -θ) 2 ; Among them, h i represents the second light intensity distribution image corresponding to the i-th sample image in the training information set, and θ represents the sample label corresponding to the current hypersurface.

[0014] As a further improvement of an embodiment of the present invention, for the same image category, along the direction from the input end to the output end of the image classification model, the area of ​​at least one preset imaging region of the sample labels corresponding to the at least two hypersurfaces becomes smaller successively.

[0015] As a further improvement of one embodiment of the present invention, the light intensity distribution image is calculated by the metasurface based on the optical propagation trajectory formula for the training samples in the training information set.

[0016] As a further improvement of an embodiment of the present invention, the optical propagation trajectory formula is:

[0017]

[0018] Among them, U out Image showing the light intensity distribution of the metasurface output, F and F -1 represent Fourier transform and inverse Fourier transform respectively, (x, y) represents the position coordinates of a point on the hypersurface, Amp(x, y) represents the amplitude of the hypersurface at point (x, y), and U in represents the initial light intensity distribution image corresponding to the training sample, k is the wave vector of the light wave, λ is the wavelength of the light wave, and i is the imaginary part.

[0019] To achieve one of the above-mentioned purposes of the invention, one embodiment of the present invention provides an image classification model, including: the image classification model includes at least two hypersurfaces arranged in sequence from the input end to the output end, and the image classification model is trained and determined according to any one of the training methods described.

[0020] To achieve one of the above-mentioned purposes of the invention, one embodiment of the present invention provides an image classification method, comprising: acquiring an image to be detected; using the image classification model to perform classification prediction on the image to be detected to obtain a light intensity distribution image to determine the image category corresponding to the image to be detected.

[0021] To achieve one of the above-mentioned purposes of the invention, one embodiment of the present invention provides an electronic device, comprising: at least one processor; a memory, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, it implements the steps of the training method described in any of the above-mentioned technical solutions, or implements the steps of the image classification method described in any of the above-mentioned technical solutions.

[0022] To achieve one of the above-mentioned purposes of the invention, one embodiment of the present invention provides a computer storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the training method described in any of the above-mentioned technical solutions are implemented, or the steps of the image classification method described in any of the above-mentioned technical solutions are implemented.

[0023] Compared with the prior art, the image classification model training method provided by the present invention trains the at least two hypersurfaces in sequence along the direction from the input end to the output end of the image classification model. The model can gradually optimize the parameters of each hypersurface so that each hypersurface can better regulate the optical signal of the input image, retain information useful for subsequent classification as much as possible, and suppress the interference of irrelevant information. This layer-by-layer training and layer-by-layer optimization method helps to maintain sufficient gradient information in the deep network, thereby avoiding the risk of gradient disappearance and improving calculation accuracy and training speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the structure of an electronic device in one embodiment of the present invention.

[0025] Figure 2 It is a structural principle diagram of an image classification model in one embodiment of the present invention.

[0026] Figure 3 It is a structural principle diagram of the hypersurface in the image classification model in one embodiment of the present invention.

[0027] Figure 4 It is a structural principle diagram of the hypersurface in the image classification model in one embodiment of the present invention.

[0028] Figure 5 It is a schematic diagram of the steps of an image classification model training method in one embodiment of the present invention.

[0029] Figure 6 It is a structural schematic diagram of the area change of a preset imaging area of ​​a sample label corresponding to a hypersurface in an image classification model in one embodiment of the present invention.

[0030] Figure 7 It is a schematic diagram of the light focusing area after the sample image is processed by the super surface in one embodiment of the present invention.

[0031] Figure 8 It is a schematic diagram of the steps of a specific example of a training method in an embodiment of the present invention.

[0032] Fig. 9 Schematic diagram of the steps of an image classification method in one embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0034] It should be noted that the term "comprise" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0035] like Figure 1 As shown, an embodiment of the present invention provides an electronic device 100.

[0036] The electronic device 100 can be any device used to train the image classification model training method provided by the present invention, can be any device used to execute the hypersurface-based forward neural network training method provided by the present invention, and can be any device used to execute the image classification method provided by the present invention.

[0037] The electronic device 100 may be a computer device, and the computer device may be a terminal device or a server. Figure 1 Only an exemplary structure of the electronic device 100 is shown, not all structures, and the electronic device 100 may be implemented as required. Figure 1 Partial or complete structure shown.

[0038] In an exemplary embodiment, the electronic device 100 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to execute the image classification model training method or image classification method provided later.

[0039] The electronic device 100 includes at least one processor 11. The image classification model training method provided by the present invention can be applied to the processor 11, or implemented by the processor 11. The image classification model training method provided by the present invention can be applied to the processor 11, or implemented by the processor 11.

[0040] The processor 11 may be an integrated circuit chip having signal processing capabilities. In the process of implementing the technical solution provided by the present invention, each step of the image classification model training method may be completed by an integrated logic circuit of hardware in the processor 11 or an instruction in the form of software. In the process of implementing the technical solution provided by the present invention, each step of the image classification method may be completed by an integrated logic circuit of hardware in the processor 11 or an instruction in the form of software.

[0041] The processor 11 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 11 may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor.

[0042] The software module may be located in a storage medium, which is located in the memory 12. The processor 11 reads the information in the memory 12 and completes the steps of the configuration method provided in the embodiment of the present invention in combination with its hardware.

[0043] The electronic device 100 includes a memory 12. The memory 12 is used to store various types of data to support the operation of the electronic device 100. Examples of such data include: any computer program for operating on a computer device.

[0044] The memory 12 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), direct memory bus random access memory (DRRAM). The memory described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0045] When the processor 11 executes the computer program stored in the memory 12, the steps of the image classification model training method in any technical solution of the present invention are executed, or the steps of the image classification method in any technical solution of the present invention are executed.

[0046] The electronic device 100 includes a user interface 14 and at least one network interface 15. The various components in the electronic device 100 are coupled together via a bus system 13. The bus system 13 is used to achieve communication connections between these components. In addition to the data bus, the bus system 13 also includes a power bus, a control bus, and a status signal bus. To simplify the description, Figure 1 The various buses are collectively labeled as a bus system 13.

[0047] The user interface 14 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0048] An embodiment of the present invention provides a computer storage medium.

[0049] The computer storage medium stores the computer program executed by the processor mentioned above, or instructions of the image classification model training method in any of the technical solutions below, or instructions of the image classification method in any of the technical solutions below.

[0050] When the processor executes the computer program, it can execute the steps of the image classification model training method in any of the technical solutions below, or execute the steps of the image classification method in any of the technical solutions below, which will not be repeated here. In addition, the description of the beneficial effects corresponding to the same method will not be repeated. For technical details not disclosed in the embodiments of the computer storage medium involved in the present invention, reference can be made to the description in the embodiments of the training method or image classification method of the present invention below.

[0051] The computer storage device may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0052] An embodiment of the present invention provides a computer program product.

[0053] The computer program product may include a computer program, which may be stored in a computer storage medium. A processor of an electronic device reads the computer program from the computer storage medium, and the processor may execute the computer program, so that the electronic device executes the description of the image classification model training method or the image classification method in any of the technical solutions below.

[0054] like Figure 2 As shown, an embodiment of the present invention provides an image classification model 200.

[0055] The image classification model 200 is used to assign sample images to specified categories or labels.

[0056] In one embodiment, the image classification model includes at least two hypersurfaces arranged in sequence from the input end to the output end.

[0057] In the embodiments of the present application, Figure 3 and Figure 4 As shown, in at least two supersurfaces of the image classification model, two adjacent supersurfaces can be independently separated or integrated. When integrated, two or more supersurface structures can be combined by thermal bonding, plasma bonding, adhesive bonding, etc.; or, another group of supersurface structures can be directly prepared on the substrate side or element side of a group of supersurface structures, for example: first, an optical film layer is formed on the substrate side or nanostructure side of a group of supersurface structures, and then the optical film layer is etched into another group of supersurface structures by photolithography.

[0058] In one embodiment, the image classification model 200 is obtained based on a training method for an image classification model provided in any of the following implementations.

[0059] Continue to refer to Figure 2 As shown, the image classification model 200 may include multiple hypersurfaces to be trained. Along the direction from the input end to the output end of the image classification model, there will be a section of freely propagating light before the training information set passes through the first hypersurface (or the first hypersurface) and after being processed by the first hypersurface.

[0060] Similarly, before the output result of the first metasurface passes through the second metasurface, and after the output result is processed by the second metasurface, there is also a period of free propagation of light. That is, the light first propagates freely, and after being processed by the current metasurface, it freely propagates to the next metasurface for further processing.

[0061] Specifically, based on the training information set, the first hypersurface is trained to determine a first loss value; based on the first loss value, the configuration parameters of the first hypersurface are updated until a preset condition is met, and a first configuration is determined; based on the first configuration, the output of the training information set corresponding to the first hypersurface is determined as a new training information set; the new training information set is input into the second hypersurface to determine a second loss value; based on the second loss value, the configuration parameters of the second hypersurface are updated until the preset condition is met, and so on until the Nth configuration of the Nth hypersurface is adjusted and determined based on the Nth loss value.

[0062] Compared with the traditional back-propagation algorithm, the present invention adopts a forward training method. Along the direction from the input end to the output end of the image classification model, each hypersurface is trained in turn, with the goal that the intensity distribution image obtained after the trained hypersurface processes the sample image and the corresponding sample label meet the preset conditions, and the configuration of the corresponding hypersurface is directly adjusted to obtain a trained hypersurface. Based on the trained hypersurface, the next hypersurface is trained using the same method until all the hypersurfaces to be trained are trained to obtain a trained image classification model.

[0063] Therefore, the hypersurfaces in the image classification model will be trained one by one, and the output of each hypersurface will be used as the input of the next hypersurface. This method simplifies the training process of the entire network, reduces computing resources, reduces the time complexity of the training method, and improves training efficiency.

[0064] by Figure 2 Taking the model structure shown in the figure as an example, the time complexity required by the traditional back propagation method is O(N 2 ), while the time required by the present invention is O(N).

[0065] In addition, this method does not require back-propagation of gradients, which can avoid the problem of difficulty in further increasing accuracy due to the increase in neural network layers in the traditional back-propagation process, thereby improving the computational accuracy of the neural network.

[0066] like Figure 5 As shown, an embodiment of the present invention provides an image classification model training method.

[0067] In one embodiment, the image classification model includes at least two hypersurfaces arranged in sequence from the input end to the output end.

[0068] In one embodiment, the training method can train the above-mentioned image classification model to obtain a trained image classification model.

[0069] In one embodiment, the training method can also be used to train an optical neural network model based on a metasurface.

[0070] In one embodiment, the optical neural network model includes a feedforward neural network.

[0071] A feedforward neural network is a neural network model that includes an input layer, a hidden layer, and an output layer. The flow of information in the network is unidirectional, that is, it flows from the input layer through one or more hidden layers to the output layer without back propagation or loop connections.

[0072] The feedforward neural network model includes several hypersurfaces to be trained.

[0073] In a feedforward neural network, only the neurons in the two adjacent layers are connected, and there is no feedback between the neurons. Each neuron can receive multiple inputs from the previous layer and has only one output to the neurons in the next layer.

[0074] In one embodiment, in an image classification model training method, multiple hypersurfaces can be used as corresponding neurons. The hypersurface can receive light signals from one or more hypersurfaces and generate transformed light signals as input to a subsequent hypersurface.

[0075] After being modeled, the metasurface can maintain the form of a physical device, and its specific configuration parameters can be trained and optimized during training. After being modeled, the metasurface can also be abstracted into the form of a mathematical model, and its model parameters can be directly trained and optimized during training.

[0076] The metasurface includes a high degree of freedom metasurface two-dimensional material, which is composed of billions of sub-wavelength meta-atoms, which can be called primitives (or micro-nano structures), and the primitives form a metasurface through a certain arrangement. The metasurface can be arranged on at least one side of a light-guiding element such as a lens, or can be directly formed on at least one side of an image sensor by etching or other means.

[0077] In one embodiment, the metasurface can be specifically configured on a sensor having a photoelectric conversion function. Specifically, a metasurface element can be formed on a sensor substrate to prepare a metasurface. In this way, the sensor having the metasurface characteristics is defined as the metasurface.

[0078] In one specific embodiment, the sensor can be specifically configured as a complementary metal oxide semiconductor (CMOS) type photosensitive element; in another specific embodiment, the sensor can also be specifically configured as a charge-coupled device (CCD) type photosensitive element, which is not specifically limited in this application. In short, a metasurface should be defined as any component having a metasurface formed by micro-nano structural units.

[0079] In one embodiment, the metasurface can be configured on a scanning lens or a shooting lens.

[0080] A metasurface can be a physical device or a simulation model constructed by a software algorithm.

[0081] In one embodiment, in the process of training for forming a forward neural network based on a metasurface, the metasurface is constructed based on an optical path simulation platform to form a metasurface imaging model.

[0082] By inputting an image to the metasurface imaging model, an output image after simulating light modulation and photoelectric conversion can be obtained. In a specific embodiment, the input is a person image, and the output image is a corresponding light intensity distribution image.

[0083] like Figure 5 As shown, the image classification model training method provided in one embodiment of the present invention includes but is not limited to the following steps.

[0084] Step S1, obtaining a training information set;

[0085] Step S2, based on the training information set, train the at least two hypersurfaces in sequence along the direction from the input end to the output end of the image classification model, so that the light intensity distribution image obtained after the trained hypersurface processes the sample image and the corresponding sample label meet preset conditions.

[0086] In this way, by training the at least two hypersurfaces in sequence along the direction from the input end to the output end of the image classification model, the model can gradually optimize the parameters of each hypersurface so that each hypersurface can better regulate the optical signal of the input image, try to retain information that is useful for subsequent classification while suppressing the interference of irrelevant information. This layer-by-layer training and layer-by-layer optimization method helps to maintain sufficient gradient information in the deep network, thereby avoiding the risk of gradient disappearance and improving calculation accuracy and training speed.

[0087] In one embodiment, the training information set includes multiple groups of training samples, each group of training samples includes a sample image and at least two sample labels corresponding to the hypersurface. For example, taking gender classification as an example, the sample labels corresponding to the hypersurface are male labels and female labels. Of course, the present invention is not limited to the two types of male and female sample labels, and may also include other labels other than male and female labels.

[0088] In one embodiment, the sample label represents the light intensity distribution of different preset imaging areas, and each of the preset imaging areas corresponds to an image category.

[0089] Specifically, the sample image is a basic data unit in the training information set. The sample image can be a digital image from various sources, such as a photo taken by a camera, a scanned document, an image generated by a computer, etc. In the above embodiment, the sample image is used as the input of the training model (image classification model).

[0090] Metasurfaces can be used as part of image processing, interacting with sample images in some way (such as physical light processing or digital simulation) to change the optical properties of the image. The metasurface learns through training how to process the input sample image to produce the desired output. For example, it is expected that the light spot in the upper right corner of the corresponding imaging area is the brightest.

[0091] The sample label is associated with each sample image and is used to indicate the image category or characteristic to which the image belongs. In the above embodiment, each group of sample images includes at least two sample labels, and these labels correspond to the light intensity distribution of different preset imaging areas. In other words, each sample image can be regarded as containing multiple imaging areas, each area has its own unique light intensity distribution characteristics, and these characteristics are characterized by corresponding feature labels.

[0092] Light intensity distribution refers to the intensity distribution of light at each point in space. In image processing, it can be associated with visual attributes such as brightness and contrast of the image. In the above embodiment, light intensity distribution refers to the specific distribution of light intensity in a preset imaging area, which can be described by a label. The goal of image classification model training is to make the light intensity distribution in a specific area of ​​the image after metasurface processing match the light intensity distribution in the sample label.

[0093] An image category refers to a specific category or classification to which an image belongs. In the above embodiment, each preset imaging area corresponds to an image category, and these image categories can be defined based on the light intensity distribution. In other words, different preset imaging areas in the sample image may belong to different image categories, depending on their light intensity distribution characteristics. By training the metasurface to recognize and process these different light intensity distributions, the classification and recognition of the image content can be achieved.

[0094] like Figure 6 As shown, the sample image corresponding to the metasurface is divided into five preset imaging areas and corresponds to five sample labels, namely, first-category sample labels, second-category sample labels, third-category sample labels, fourth-category sample labels, and other-category sample labels (or, labels other than the first, second, third, and fourth-category sample labels). These sample labels characterize the light intensity distribution images of different preset imaging areas and are used to describe the content or features of the sample image in each preset imaging area. Each of the preset imaging areas corresponds to an image category.

[0095] The sample image is processed by several hypersurfaces in turn. After each hypersurface processing, the features of the sample image will be further extracted and converted for subsequent classification tasks. After the last hypersurface processing, each predicted imaging area will generate a corresponding output; the output can be a light field intensity image, and the output of each preset imaging area is compared with the corresponding preset sample label to predict whether the content of each preset imaging area is consistent with a certain sample label; when the output of a certain preset imaging area and the corresponding sample label meet the preset conditions, the sample label category corresponding to the preset imaging area is the image category of the sample image.

[0096] For example, continue to refer to Figure 6As shown, if the light intensity distribution of the upper left corner of the sample image on the CMOS after the sample image is processed by the image classification model is consistent with the light intensity distribution of the corresponding first-class sample label, then the image category corresponding to the sample image is the image category represented by the first-class sample label. In addition, after the sample image is processed by multiple metasurfaces in sequence, the area of ​​at least one preset imaging area of ​​the corresponding sample label becomes smaller in sequence.

[0097] In one embodiment, the training samples may be obtained based on a Monte Carlo random method and randomly assigned sample labels.

[0098] In this way, by using the Monte Carlo random method and randomly assigning sample labels, a large number of samples can be automatically and quickly generated, simplifying the data preparation process and improving training efficiency.

[0099] The Monte Carlo random method, also known as a statistical simulation method or a random sampling method, is a numerical calculation method based on a random simulation method and a probability and statistical theory method. It is a method that uses random numbers or pseudo-random numbers to solve many calculation problems.

[0100] In a specific embodiment, the sample images randomly generated based on Monte Carlo are noises, and the image information does not include the image features of males or females. Male and female labels are randomly assigned to them to form training samples.

[0101] In one embodiment, the image classification model training method may include a step of initializing the hypersurface.

[0102] In a specific embodiment, when initializing the metasurface, the process may include: setting initial parameters according to the application scenario. The initial parameters include at least one of the waveform of the light source, the wavelength of the light source, the material of the metasurface structure substrate, the size of the metasurface structure substrate, the distance between the metasurface structure and the light source, the distance between the metasurface structure and the image sensor (or imaging surface), and imaging parameters (e.g., imaging resolution, pixel size).

[0103] In a specific embodiment, when initializing the wavelength of the light source, the process may include: determining corresponding training data based on a non-broadband light source (without color) or a broadband light source (with color).

[0104] In a specific embodiment, based on a non-bandwidth light source, a corresponding grayscale image is used as training data.

[0105] In a specific embodiment, based on a broadband light source, corresponding color images are used as training data.

[0106] In one embodiment, the hypersurface-based forward neural network training method may include a step of initializing the training data in the training information set.

[0107] In a specific embodiment, the training data is a sample image, and when initializing the training data, the following may be included: adjusting the image parameters of the sample image according to the aperture position of the camera lens or the scanning lens. The image parameters include at least one of the size of the sample image, the cropping area of ​​the sample image, and the color configuration of the sample image.

[0108] In this way, the size of the image captured by an actual camera at different aperture settings can be simulated by adjusting or cropping the sample image to ensure that the size and range of the sample image matches the aperture setting.

[0109] In a specific embodiment, the sample image is changed into a grayscale image by setting the grayscale image to match the type of light source actually used.

[0110] Specifically, according to the application scenario, the image is captured based on a light source with a specific wavelength (such as 940nm) to determine the grayscale image. Therefore, changing the sample image to a grayscale image can better simulate the actual conditions, because the 940nm light source is a broadband light source without color.

[0111] Of course, the present invention does not necessarily use grayscale images, and the sample image can be adjusted accordingly according to the type of light source actually used.

[0112] like Figure 7 As shown in Figure 1, the metasurface is used to classify images. The input of the metasurface is a sample image, and the corresponding light signal is input into the metasurface for modulation, so that it can identify the gender characteristics in the image, and by adjusting the metasurface configuration, the light undergoes specific phase changes during propagation, and these phase changes cause the light to converge at a specific position on the CMOS sensor.

[0113] Continue to refer to Figure 7 As shown in the figure, if the gender feature in the sample image is male, the metasurface will focus the light on the upper left corner of the CMOS sensor, forming a light spot corresponding to the "male" classification result. Similarly, if the gender feature in the sample image is female, the light will be focused on the lower right corner of the CMOS sensor, forming a light spot corresponding to the "female" classification result.

[0114] The loss value is used to measure the ability of the image classification model to distinguish sample labels in sample images.

[0115] In one embodiment, the loss value is positively correlated with the discrimination ability, that is, the higher the loss value, the stronger the image classification model's ability to discriminate sample labels.

[0116] In the first embodiment, step S2 of the present invention may include but is not limited to the following steps.

[0117] Step S2': training a first hypersurface to be trained based on the training information set, wherein the first hypersurface to be trained is the first hypersurface close to the input end of the image classification model.

[0118] In the second embodiment, step S2 of the present invention may include but is not limited to the following steps.

[0119] Step S2", training the second hypersurface to be trained based on the first light intensity distribution output by the previous hypersurface of the second hypersurface to be trained, where the second hypersurface to be trained is a hypersurface other than the first hypersurface to be trained.

[0120] In this way, by using the output of the previous hypersurface as the input of the current hypersurface for training, the layer-by-layer transmission and deepening of features can be achieved. This layer-by-layer training method helps the model gradually build a feature hierarchy structure from low-level to high-level, thereby improving the accuracy of image classification.

[0121] In one embodiment, the above two embodiments can be used in combination. That is, if the current hypersurface to be trained is the first hypersurface close to the input end of the image classification model, the first hypersurface is trained based on the training information set; if the current model to be trained is not the first hypersurface, the output of the previous hypersurface is used as a new training information set to train the current hypersurface.

[0122] In a specific embodiment, after the training sample is processed by the super surface, a light intensity distribution image is output.

[0123] like Figure 8 As shown, in one embodiment, the training process for each hypersurface in step S2 of the present invention may include but is not limited to the following steps.

[0124] Step S21, based on the training sample or the first light intensity distribution image corresponding to the current metasurface, a second light intensity distribution image obtained by processing the current metasurface;

[0125] Step S22, calculating a loss value of the second light intensity distribution image based on a loss function;

[0126] Step S23, updating the configuration parameters of the current hypersurface based on the loss value until a preset condition is met.

[0127] The first light intensity distribution image and the second light intensity distribution image are both actual signal intensity distributions output after the input image is modulated by the metasurface on the amplitude and / or phase of the light signal.

[0128] In one embodiment, if the current hypersurface is the first hypersurface close to the input end of the image classification model, a second light intensity distribution image is obtained by processing the first hypersurface based on the training sample; a loss value of the second light intensity distribution image is calculated based on a loss function; and the configuration parameters of the first hypersurface are updated based on the loss value until a preset condition is met.

[0129] In this embodiment, the loss value is determined by the second light intensity distribution image of the training sample after being processed by the first hypersurface and the corresponding sample label, so that the trained first hypersurface can be obtained.

[0130] In one embodiment, if the current hypersurface is not the first hypersurface close to the input end of the image classification model, a second light intensity distribution image is obtained based on the first light intensity distribution image corresponding to the current hypersurface using the current hypersurface processing; a loss value of the second light intensity distribution image is calculated based on a loss function; and configuration parameters of the current hypersurface are updated based on the loss value until preset conditions are met.

[0131] In this embodiment, the first light intensity distribution image is determined based on the output result of a previously trained hypersurface of the current hypersurface, so that the trained current hypersurface can be continuously obtained.

[0132] In one embodiment, the light intensity distribution image is calculated by the metasurface based on the optical propagation trajectory formula on the training samples in the information set.

[0133] In a specific embodiment, the optical propagation trajectory formula is:

[0134]

[0135] Among them, U out Image showing the light intensity distribution of the metasurface output, F and F -1 represent Fourier transform and inverse Fourier transform respectively, (x, y) represents the position coordinates of a point on the hypersurface, Amp(x, y) represents the amplitude of the hypersurface at point (x, y), and U im represents the initial light intensity distribution image corresponding to the training sample, k is the wave vector of the light wave, λ is the wavelength of the light wave, and i is the imaginary part.

[0136] In the optical propagation trajectory formula, the initial light intensity distribution of the training sample is first modulated in the time domain in amplitude and phase based on the metasurface to obtain the first light wave information; the modulated first light wave information is subjected to Fourier transform to convert its light wave information from the time domain to the frequency domain; the result of the Fourier transform is subjected to inverse Fourier transform to convert its light wave information from the frequency domain back to the time domain, thereby obtaining the light intensity distribution image after the metasurface modulation.

[0137] In this way, the computation speed can be increased because the modulation of amplitude and phase can be processed faster in the frequency domain than in the time domain.

[0138] In one embodiment, if the training sample is a sample image, the initial light intensity distribution of the sample image can be understood as the light intensity (or brightness) distribution at different positions or pixels in the sample image. In one embodiment, the grayscale value can be used for measurement, and the higher the grayscale value, the greater the light intensity and the higher the brightness at that position.

[0139] In one embodiment, the metasurface is a planar structure formed by arranging a plurality of sub-wavelength-sized micro-nano structural units. The configuration parameters of the metasurface include at least one of the geometric structure, size, and spatial arrangement of the micro-nano structural units on the metasurface. The geometric structure of the micro-nano structural unit includes but is not limited to a circle, a cross, a regular polygon, a rectangle, and any combination thereof. The size of the micro-nano structural unit includes but is not limited to a radius, a side length, a height, etc. The spatial arrangement of the micro-nano structural unit includes an arrangement period, a duty cycle, etc.

[0140] In a specific embodiment, the training samples in the training information set are input into the image classification model for training, and the initial light intensity distribution image in the sample image is processed by the hypersurface to obtain the light intensity distribution image. The loss value of the light intensity distribution image is calculated based on the loss function, and the radius of the primitive on the hypersurface is adjusted; it is determined whether the loss value is within the defined difference range; if so, the adjusted radius of the hypersurface primitive is determined; if not, the radius of the primitive on the hypersurface is continued to be adjusted.

[0141] In a specific embodiment, the difference in loss values ​​is positively correlated with the parameter configuration of the hypersurface. In other words, the greater the difference between the intensity distribution image and the intensity distribution represented by the sample label (i.e., the expected intensity distribution), the greater the adjustment of the primitive radius on the hypersurface, and vice versa.

[0142] In a specific embodiment, the adjustment of the primitive radius on the hypersurface includes positive adjustment (increasing the primitive radius) or negative adjustment (decreasing the primitive radius), that is, there is no specific restriction on the direction of adjustment of the primitive radius.

[0143] In one embodiment, the loss value of the second light intensity distribution is calculated based on the following loss function: G = ∑relu(h i -θ) 2 ; Among them, h i represents the second light intensity distribution corresponding to the i-th training sample in the training information set, and θ represents the sample label corresponding to the current hypersurface.

[0144] Among them, the relu function (Rectified Linear Unit) is an activation function used to increase the nonlinearity of the neural network. i -θ) is greater than 0, the output (h i -θ); when (h i -θ) is less than 0, the output is 0.

[0145] In this embodiment, relu(h i -θ) 2 This means that the training data whose light intensity distribution after the super-surface treatment exceeds the light intensity distribution θ represented by the sample label has a non-zero contribution to the loss value, and all of them are non-negative, which makes the loss value convex, making it easy to find the global optimal solution using standard optimization algorithms.

[0146] In one embodiment, for the same image category, along the direction from the input end to the output end of the image classification model, the area of ​​at least one preset imaging region of the sample label corresponding to the at least two hypersurfaces becomes smaller in sequence, which can be referred to as Figure 6 shown.

[0147] In this way, as the image classification model processes more deeply, each layer of the hypersurface further processes and transforms the light intensity distribution. In this process, features related to the classification task will be strengthened, while features that are less relevant to the classification task will be suppressed or ignored. Therefore, the smaller area means that the model is gradually removing redundant information, which helps to improve the accuracy of model classification.

[0148] In one embodiment, the first configuration h i The degree of match between the light intensity distribution image after the metasurface processing and the light intensity distribution image of the preset imaging area represented by the sample label (i.e., the expected light intensity distribution) is determined by continuously adjusting the configuration parameters of the metasurface so that the position and / or intensity of the light intensity distribution image after the metasurface processing and the expected light intensity distribution image corresponding to the sample label are more closely matched.

[0149] like Fig. 9 As shown, an embodiment of the present invention provides an image classification method.

[0150] The image classification method is used to classify data including but not limited to images. In one embodiment, the image classification method is used to classify images to be detected. Based on this, the image classification method of the present invention includes the following steps.

[0151] Step M1, executing the image classification model training method described in any of the above technical solutions to obtain a trained image classification model;

[0152] Step M2, obtaining the image to be detected, using the image classification model to perform classification prediction on the image to be detected, and obtaining a light intensity distribution image to determine the image category corresponding to the image to be detected.

[0153] In this way, the trained image classification model can be used to quickly classify the images to be detected, significantly reducing processing time and improving classification efficiency.

[0154] In one embodiment, step M2 may specifically include but is not limited to the following steps.

[0155] Step M21, predicting the light intensity distribution area of ​​the image to be detected based on the image classification model; the light intensity distribution area at least includes the target area;

[0156] Step M22: determining corresponding classification results based on the light intensity distribution images of the target area and other areas.

[0157] In one embodiment, the image to be detected is unsupervised data, that is, unannotated or labeled data. In this embodiment, it is classified by a trained image classification model to generate an image category label.

[0158] In one embodiment, the image to be detected is fully supervised data, that is, there is image category label data. In this embodiment, it is classified by a trained image classification model to determine and / or update the label. When the prediction result does not match the known image category label data, the image category label data is updated.

[0159] In a specific embodiment, the light intensity distribution area includes a first target area and a second target area; based on the light intensity distribution images of the first target area and the second target area, a corresponding classification result is determined. The final classification result is determined based on the degree of matching between the light intensity distribution images of the two target areas and the predicted light intensity distribution images corresponding to the preset classification labels.

[0160] For example, unlabeled male and female images or corresponding raw data are obtained, and the male and female images or raw data are predicted based on the super lens designed based on the super surface theory and the trained image classification model to predict the target area where the light converges. The category is judged based on the light field intensity perceived by the CMOS photoreceptor in a certain area. For example, if the light intensity in the male area is greater than that in the female area or other areas, the sample image is predicted to be male.

[0161] In summary, the image classification model and its training method, image classification method, electronic device and computer storage medium provided by the present invention. By training the at least two hypersurfaces in sequence along the direction from the input end to the output end of the image classification model, the model can gradually optimize the parameters of each hypersurface, so that each hypersurface can better regulate the optical signal of the input image, try to retain information useful for subsequent classification while suppressing the interference of irrelevant information, and this layer-by-layer training and layer-by-layer optimization method helps to maintain sufficient gradient information in the deep network, thereby avoiding the risk of gradient disappearance and improving calculation accuracy and training speed.

[0162] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0163] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for training an image classification model, characterized in that: The image classification model includes at least two hypersurfaces arranged in sequence from the input end to the output end, and the training method includes: Acquire a training information set, the training information set comprising a plurality of groups of training samples, each group of the training samples comprising a sample image and at least two sample labels corresponding one-to-one to the metasurface, wherein the sample labels represent light intensity distribution images of different preset imaging areas, and each of the preset imaging areas corresponds to an image category; Based on the training information set, the at least two hypersurfaces are trained in sequence along the direction from the input end to the output end of the image classification model, so that the light intensity distribution image obtained after the trained hypersurface processes the sample image and the corresponding sample label meet preset conditions.

2. The training method according to claim 1, characterized in that: The step of training the at least two hypersurfaces in sequence along a direction from an input end to an output end of the image classification model based on the training information set includes: Based on the training information set, training a first hypersurface to be trained, where the first hypersurface to be trained is a first hypersurface close to an input end of the image classification model; and / or, Based on the first light intensity distribution image output by a previous hypersurface of the second hypersurface to be trained, the second hypersurface to be trained is trained, and the second hypersurface to be trained is a hypersurface other than the first hypersurface to be trained.

3. The training method according to claim 2, characterized in that: The method further comprises: Based on the first light intensity distribution image corresponding to the training sample or the current metasurface, obtaining a second light intensity distribution image by processing the current metasurface; Calculating a loss value of the second light intensity distribution image based on a loss function; The configuration parameters of the current hypersurface are updated based on the loss value until the preset conditions are met.

4. The training method according to claim 3, characterized in that: The calculating the loss value of the second light intensity distribution image based on the loss function includes: The loss value of the second light intensity distribution image is calculated based on the following loss function: G=∑relu(h i -i) 2 ; Among them, h i represents the second light intensity distribution image corresponding to the i-th training sample in the training information set, and θ represents the sample label corresponding to the current hypersurface.

5. The training method according to claim 1, characterized in that: For the same image category, along the direction from the input end to the output end of the image classification model, the area of ​​at least one preset imaging region of the sample labels corresponding to the at least two hypersurfaces becomes smaller in sequence.

6. The training method according to claim 2, characterized in that: The light intensity distribution image is obtained by calculating the training samples in the training information set based on the optical propagation trajectory formula of the metasurface.

7. The training method according to claim 6, characterized in that: The optical propagation trajectory formula is: Among them, U out Image showing the light intensity distribution of the metasurface output, F and F -1 denote Fourier transform and inverse Fourier transform respectively, (x, y) denotes the position coordinates of a point on the hypersurface, Amp(x, y) denotes the amplitude of the hypersurface at the point (x, y), and U in represents the initial light intensity distribution image corresponding to the training sample, k is the wave vector of the light wave, λ is the wavelength of the light wave, and i is the imaginary part.

8. An image classification model, characterized in that: include: The image classification model comprises at least two hypersurfaces arranged in sequence from the input end to the output end, and the image classification model is trained and determined according to the training method according to any one of claims 1-7.

9. An image classification method, characterized in that: include: Acquire the image to be detected; The image classification model described in claim 8 is used to perform classification prediction on the image to be detected to obtain a light intensity distribution image to determine the image category corresponding to the image to be detected.

10. An electronic device comprising: at least one processor; A memory storing a computer program executable on the processor, wherein the processor implements the steps of the training method according to any one of claims 1 to 7, or implements the steps of the image classification method according to claim 9 when executing the program.

11. A computer storage medium, wherein the computer readable storage medium stores a computer program, characterized in that: When the computer program is executed by a processor, the steps of the training method according to any one of claims 1 to 7 are implemented, or the steps of the image classification method according to claim 9 are implemented.

Images