Picture generation method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a picture generation method and device, equipment, a storage medium and a program product. The method comprises the following steps: obtaining a first initial picture through a real camera; wherein the real camera comprises a differentiable lens encoder in a pre-trained picture generation model; the picture generation model further comprises a decoder; the picture generation model is obtained by performing end-to-end training through a depth optical tolerance combined point diffusion loss function and a light spot loss function in advance; and decoding the first initial picture through the decoder to obtain a first target picture. According to the embodiment of the invention, through the mode of generating the first target picture by using the picture generation model obtained by combining the depth optical tolerance with the point diffusion loss function and the light spot loss function to carry out end-to-end training, the error between a differentiable lens encoder (lens) simulation design stage and an actual manufacturing stage can be reduced; and the picture generation effect is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, storage medium and program product for generating an image. Background Art

[0002] There are often errors in the optical imaging system between the simulation design stage and the actual manufacturing stage, that is, the robustness of the optical imaging system is poor, resulting in a large difference in the image effects generated in the simulation design stage and the actual manufacturing stage, and a poor user experience.

[0003] However, most of the current research is based on the assumption that there is a perfect match between the simulation design stage and the actual manufacturing stage. How to reduce the error between the simulation design stage and the actual manufacturing stage has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides a method, device, equipment, storage medium and program product for generating an image, which can reduce the error between the simulation design stage and the actual manufacturing stage and improve the effect of image generation.

[0005] In a first aspect, an embodiment of the present application provides a method for generating an image, comprising: obtaining a first initial image through a real camera; wherein the real camera includes a differentiable lens encoder in a pre-trained image generation model; the image generation model also includes a decoder; the image generation model is pre-trained end-to-end through deep optical tolerance combined with a point spread loss function and a spot loss function; the first initial image is decoded by the decoder to obtain a first target image.

[0006] In a second aspect, an embodiment of the present application further provides an image generating device, comprising: a first initial image obtaining module, used to obtain a first initial image through a real camera; wherein the real camera includes a differentiable lens encoder in a pre-trained image generation model; the image generation model also includes a decoder; the image generation model is pre-trained end-to-end through deep optical tolerance combined with a point spread loss function and a spot loss function; a decoding module, used to decode the first initial image through the decoder to obtain a first target image.

[0007] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image generation method as described in the embodiment of the present application.

[0008] In a fourth aspect, an embodiment of the present application further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the image generation method as described in the embodiment of the present application.

[0009] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the image generation method as described in the embodiment of the present application.

[0010] The technical solution of the embodiment of the present application is to obtain a first initial image through a real camera; wherein the real camera includes a differentiable lens encoder in a pre-trained image generation model; the image generation model also includes a decoder; the image generation model is pre-trained end-to-end through deep optical tolerance combined with point spread loss function and spot loss function; the first initial image is decoded by the decoder to obtain a first target image. The disclosed embodiment generates the first target image by using a picture generation model obtained by end-to-end training with deep optical tolerance combined with point spread loss function and spot loss function, which can reduce the error between the simulation design stage and the actual manufacturing stage of the differentiable lens encoder (lens) and improve the effect of picture generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0012] Figure 1 A schematic diagram of a picture generation method provided in an embodiment of the present application;

[0013] Figure 2 A schematic diagram of the structure of a picture generating device provided in an embodiment of the present application;

[0014] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0016] It should be understood that the various steps recorded in the method implementation of the present disclosure can be performed in different orders and / or in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect. The term "including" and its variations used herein are open inclusions, that is, "including but not limited to". It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless otherwise clearly stated in the context, it should be understood as "one or more". It is understandable that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of the corresponding laws, regulations and relevant provisions.

[0017] Figure 1 The following is a flow chart of a method for generating an image provided by an embodiment of the present application. The method can be executed by an image generating device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC, or a server. Figure 1 As shown, the method includes:

[0018] S110: Obtain a first initial picture through a real camera.

[0019] The real camera includes a differentiable lens encoder in a pre-trained image generation model; the image generation model also includes a decoder; the image generation model is pre-trained end-to-end through deep optical tolerance combined with a point spread loss function and a spot loss function.

[0020] In this embodiment, the real camera can be any type of camera as long as it has a shooting function. The real camera includes a differentiable lens encoder and optical parameters of the differentiable lens encoder in a pre-trained picture generation model. That is, the differentiable lens encoder and optical parameters of the differentiable lens encoder in actual application are the differentiable lens encoder and optical parameters of the differentiable lens encoder trained in the training stage. The differentiable lens encoder can be understood as lens hardware.

[0021] Among them, the deep optical tolerance can be understood as the inevitable error existing in the processing or assembly process of the optical imaging system (such as a lens), that is, the error between the optical imaging system (such as a lens) in the simulation design stage and the actual manufacturing stage. The deep optical tolerance includes tilt error, eccentric error, center thickness error and curvature radius error. Among them, the first initial picture is a picture taken by a real camera.

[0022] Among them, the point spread loss function is used to determine the loss value of the point spread function (or point spread function graph) based on the similarity between the point spread function (or point spread function graph) of the differentiable lens encoder and the preset point spread function (or preset point spread function graph), and the spot loss function is used to determine the average spot size of the light spot array diagram of the differentiable lens encoder in the entire field of view (that is, the average spot size of all grids), and used as the loss value of the light spot array diagram.

[0023] Among them, end-to-end can be understood as a training method for coordinated optimization of software and hardware parameters.

[0024] It should be noted that during the training phase, the image generation model takes deep optical tolerance into account during the ray tracing process of the differentiable lens encoder, ensuring that the obtained point spread function diagram and light spot array diagram are the real results taking into account deep optical tolerance.

[0025] S120. Decode the first initial picture by using the decoder to obtain a first target picture.

[0026] The decoder may be understood as any neural network, such as a convolutional neural network of a U-Net architecture. In this embodiment, the first initial image may be decoded by the decoder to restore a high-quality image. The first target image is an image obtained by decoding the first initial image.

[0027] The first target image obtained by the solution provided in this embodiment (using deep optical tolerance optimization) has higher clarity and better effect than the image obtained without using deep optical tolerance optimization.

[0028] The technical solution of the embodiment of the present application is to obtain a first initial image through a real camera; wherein the real camera includes a differentiable lens encoder in a pre-trained image generation model; the image generation model also includes a decoder; the image generation model is pre-trained end-to-end through deep optical tolerance combined with point spread loss function and spot loss function; the first initial image is decoded by the decoder to obtain a first target image. The disclosed embodiment generates the first target image by using a picture generation model obtained by end-to-end training using deep optical tolerance combined with point spread loss function and spot loss function, which can reduce the error between the simulation design stage and the actual manufacturing stage and improve the effect of picture generation.

[0029] Optionally, the training method of the image generation model includes: obtaining a training picture set; randomly sampling the depth optical tolerance through a tolerance sampler during the ray tracing process of the differentiable lens encoder; transforming the spatial position and curvature of the differentiable lens encoder based on the depth optical tolerance; obtaining a point spread function atlas and a light spot column atlas corresponding to the training picture set through the differentiable lens encoder based on the training picture set and a ray tracing method; determining a simulation picture set based on the point spread function atlas and the training picture set; decoding the simulation picture set through the decoder to obtain a second target picture set; determining a target loss value based on the training picture set, the second target picture set, the point spread function atlas and the light spot column atlas; training the image generation model based on the loss value until a training stop condition is met to obtain a trained image generation model.

[0030] The training image set can be any public image dataset, and the images can include various contents, such as scenic spots, people, etc.

[0031] In this embodiment, a tolerance sampler can be used to randomly sample the depth optical tolerance that obeys the Gaussian distribution during the forward ray tracing process of the differentiable lens encoder, that is, the depth optical tolerance is explicitly modeled in the ray tracing process (depth optical tolerance perturbation is performed) to accurately simulate the random errors of the differentiable lens encoder (lens hardware) in actual applications, thereby accurately simulating the influence of the depth tolerance on the imaging effect.

[0032] In this embodiment, the depth optical tolerance can be converted into the spatial position transformation and curvature transformation of the lens structure corresponding to the differentiable lens encoder, so that the depth optical tolerance can be modeled in the process of ray tracing, that is, after adjusting the spatial position and curvature of the differentiable lens encoder according to the depth optical tolerance, the ray tracing technology is used again to obtain key indicators reflecting the imaging properties of the differentiable lens encoder after considering the depth optical tolerance, such as the point spread function diagram and the light spot array diagram.

[0033] The training picture set includes a plurality of training pictures. The point spread function atlas set includes a plurality of point spread function graphs. The light spot column atlas set includes a plurality of light spot column graphs. The simulation picture set includes a plurality of simulation pictures. The second target picture set includes a plurality of second target pictures. One training picture corresponds to one spread function graph, one training picture corresponds to one light spot column graph, one spread function graph corresponds to one simulation picture, and one simulation picture corresponds to one second target picture.

[0034] Among them, the light spot array diagram and the spread function diagram can be understood as key indicators reflecting the imaging properties of the differentiable lens encoder obtained by ray tracing technology. Specifically, the ray tracing technology obtains the point spread function diagram within the corresponding grid range by tracing the point light source of the object plane grid.

[0035] The simulated image can be understood as an image obtained by performing a convolution operation on the corresponding training image through the corresponding point spread function graph and applying noise. The second target image can be understood as an image obtained by decoding the simulated image through a decoder.

[0036] Among them, the target loss value includes image loss value, spot loss value and point spread loss value.

[0037] In this embodiment, the image loss value may be obtained through the training image set and the second target image set, the point spread loss value may be obtained through the point spread function atlas, and the spot loss value may be obtained through the spot array atlas.

[0038] In this embodiment, there is no restriction on the training stop condition. For example, training is stopped when the number of iterations is greater than or equal to the set number, or training is stopped when the target loss value is less than or equal to the set loss value.

[0039] Optionally, a simulation picture set is determined based on the point spread function atlas and the training picture set, including: for each training picture, based on the point spread function atlas corresponding to the training picture, performing a spatially variable convolution operation on the training picture to obtain a second initial picture; and applying Gaussian noise to the second initial picture to obtain a simulation picture corresponding to the training picture.

[0040] The second initial image may be an image obtained by convolving the corresponding training image through the point spread function image. In this embodiment, each training image will be subjected to the same operation as follows: using the point spread function image corresponding to the training image, the training image is subjected to a spatially variable convolution operation to obtain the second initial image. A Gaussian noise of a set proportion (such as 1%) is applied to the second initial image to obtain a simulated image corresponding to the training image.

[0041] Optionally, the image generation model is trained based on the loss value until a training stop condition is met to obtain a trained image generation model, including: determining an image loss value based on the training image set and the second target image set; determining a point spread loss value based on the similarity between each point spread function graph in the point spread function graph set and a preset point spread function graph through the point spread loss function; wherein the preset point spread function graph corresponding to each point spread function graph is the same; determining a spot loss value based on the spot column graph set through the spot loss function; training the image generation model based on the image loss value, the point spread loss value and the spot loss value until a training stop condition is met to obtain a trained image generation model.

[0042] In this embodiment, the image loss value can be determined based on the training image set and the second target image set by an image quality loss function. The image quality loss function includes a mean square error loss function (MSE Loss), a total variation loss function (TV Loss) and a perceptual loss function (such as VGG Loss). The mean square error loss function is used to calculate the average of the squares of the pixel differences between the training image and the second target image. The total variation loss function is used to calculate the smooth difference between the training image and the second target image. The perceptual loss function is used to calculate the difference between the training image and the second target image in the perceptual space to measure the image quality.

[0043] In this embodiment, the weight coefficients between the mean square error loss function, the total variation loss function, the perception loss function, the point spread loss function, and the spot loss function may vary depending on the lens structure. Specifically, the weight coefficients of each loss function can be determined by testing the size of the returned gradient. For example, the weight of the point spread loss function is 0.05, the weight of the spot loss function is 0.2, the weight of the mean square error loss function is 1.0, the weight of the total variation loss function is 0.001, and the weight of the perception loss function is 0.01.

[0044] In this embodiment, the average loss value of the training picture set and the second target picture set is used as the picture loss value of the second target picture set (each training picture and the corresponding second target picture have a loss value).

[0045] In this embodiment, the average value of the similarity between each point spread function graph in the point spread function graph set and the preset point spread function graph is determined by the point spread loss function, and the average value of the similarity is subtracted from 1 to obtain the point spread loss value of the point spread function graph set. Among them, the more similar the similarity between the point spread function graph and the preset point spread function graph, the smaller the loss value between the point spread function graph and the preset point spread function graph, the less similar the similarity between the point spread function graph and the preset point spread function graph, the greater the loss value between the point spread function graph and the preset point spread function graph. By such constraints, the range of variation of the point spread function graph is limited, and the resistance of optical hardware to depth tolerance in deep optical design can be enhanced (the so-called resistance is the technical solution provided by the present invention. In practical applications, it will not be affected by the depth tolerance. Even if it is affected by the depth tolerance, the influence on the imaging properties does not change much, and the generated first target image still has a good effect, that is, robustness). Among them, the preset point spread function graph corresponding to each point spread function graph is the same; the preset point spread function graph can be understood as a point spread function graph under ideal conditions (not affected by the depth optical tolerance).

[0046] In this embodiment, the spot loss value is directly calculated based on the spot array atlas by using the spot loss function, and the smaller the spot loss value, the better.

[0047] In this embodiment, the image generation model is trained based on the image loss value of the second target image set, the point spread loss value of the point spread function atlas, and the spot loss value of the spot column atlas until the training stop condition is met to obtain a trained image generation model. During the training process, the optical parameters of the differentiable lens encoder and the parameters of the decoder are optimized, that is, the software and hardware parameters are optimized in a coordinated manner.

[0048] In this embodiment, during the training phase, the image quality loss function, the point spread loss function, and the spot loss function are used in coordination with each other, so as to ensure the effect of depth tolerance optimization while maintaining the performance of the overall design.

[0049] In this embodiment, during the training phase, the deep optical tolerance is explicitly integrated into the ray tracing (i.e., during the imaging simulation process), while maintaining the differentiable properties of deep optics (i.e., in the entire forward ray tracing, the differentiability of the operation is guaranteed so that the gradient return can jointly optimize the software and hardware parameters).

[0050] Optionally, the spot loss value is determined based on the spot array set through the spot loss function, including: for each spot array, the average spot size of the multiple grids is determined through the spot loss function as the loss value of the spot array.

[0051] Each light spot array diagram includes a plurality of grids; each grid includes a light spot.

[0052] In this embodiment, for each light spot array diagram, the light spot loss function is used to determine the average light spot size of the multiple grids as the loss value of the light spot array diagram, so that the loss values ​​of all light spot array diagrams can be obtained, and the average value is taken as the spot loss value of the light spot array diagram set.

[0053] This embodiment proposes an end-to-end, software-hardware collaborative depth tolerance optimization method to improve the resistance of deep optics to depth tolerance from both software and hardware aspects, thereby improving the manufacturing yield of the design. The end-to-end collaborative optimization of deep optics tolerance can maintain the encoding and decoding relationship between software and hardware, while reducing the sensitivity of deep optics tolerance.

[0054] In this embodiment, by explicitly modeling the deep optical tolerance in the ray tracing process (i.e., the imaging simulation process), the error between the simulation design stage and the actual manufacturing stage is greatly reduced, so that the deep optical optimization process (training process) has the ability to perceive tolerances. That is, by explicitly modeling the deep optical tolerance, the impact of the deep optical tolerance is incorporated into the imaging simulation process, and the end-to-end deep optical tolerance optimization of deep optics can be achieved through the diffusion loss function and the spot loss function, thereby reducing the error between the simulation design stage and the actual manufacturing stage.

[0055] Figure 2 A schematic diagram of the structure of a picture generating device provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the device includes: a first initial image acquisition module 210 and a decoding module 220; the first initial image acquisition module 210 is used to obtain a first initial image through a real camera; wherein the real camera includes a differentiable lens encoder in a pre-trained image generation model; the image generation model also includes a decoder; the image generation model is pre-trained end-to-end through deep optical tolerance combined with a point spread loss function and a spot loss function;

[0056] The decoding module 220 is configured to decode the first initial picture through the decoder to obtain a first target picture.

[0057] The technical solution of the embodiment of the present application is to obtain a first initial picture through a real camera through a first initial picture acquisition module; wherein the real camera includes a differentiable lens encoder in a pre-trained picture generation model; the picture generation model also includes a decoder; the picture generation model is pre-trained end-to-end through deep optical tolerance combined with a point spread loss function and a spot loss function; the first initial picture is decoded by the decoder through a decoding module to obtain a first target picture. In the embodiment of the present disclosure, the first target picture is generated by using a picture generation model obtained by end-to-end training using deep optical tolerance combined with a point spread loss function and a spot loss function, which can reduce the error between the simulation design stage and the actual manufacturing stage and improve the effect of picture generation.

[0058] Optionally, the above-mentioned device also includes a training module, which is specifically used to: obtain a training picture set; randomly sample the depth optical tolerance through a tolerance sampler during the ray tracing process of the differentiable lens encoder; transform the spatial position and curvature of the differentiable lens encoder based on the depth optical tolerance; obtain a point spread function atlas and a light spot column atlas corresponding to the training picture set through the differentiable lens encoder based on the training picture set and a ray tracing method; determine a simulation picture set based on the point spread function atlas and the training picture set; decode the simulation picture set through the decoder to obtain a second target picture set; determine a target loss value based on the training picture set, the second target picture set, the point spread function atlas and the light spot column atlas; train the picture generation model based on the loss value until a training stop condition is met to obtain a trained picture generation model.

[0059] The depth optical tolerance is a tolerance that obeys Gaussian distribution; the depth optical tolerance includes tilt error, eccentricity error, center thickness error and curvature radius error.

[0060] The point spread function atlas includes a plurality of point spread function atlases; the training picture set includes a plurality of training pictures; and the simulation picture set includes a plurality of simulation pictures.

[0061] Optionally, the training module is specifically used to: for each training image, based on a point spread function graph corresponding to the training image, perform a spatially variable convolution operation on the training image to obtain a second initial image; and apply Gaussian noise to the second initial image to obtain a simulated image corresponding to the training image.

[0062] Optionally, the training module is also used to: determine the image loss value based on the training image set and the second target image set; determine the point spread loss value based on the similarity between each point spread function graph in the point spread function graph set and a preset point spread function graph through the point spread loss function; wherein the preset point spread function graph corresponding to each point spread function graph is the same; determine the spot loss value based on the spot column graph set through the spot loss function; train the image generation model based on the image loss value, the point spread loss value and the spot loss value until the training stop condition is met to obtain the trained image generation model.

[0063] The light spot array atlas set includes a plurality of light spot arrays; each light spot array includes a plurality of grids; and each grid includes a light spot.

[0064] Optionally, the training module is further used to: for each light spot array diagram, determine the average light spot size of the multiple grids through the light spot loss function as the loss value of the light spot array diagram.

[0065] The image generation device provided in the embodiment of the present application can execute the image generation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0066] Figure 3 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0067] like Figure 3As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the random access memory (RAM) 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the read-only memory (ROM) 12 and the random access memory (RAM) 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0068] A number of components in the electronic device 10 are connected to an input / output (I / O) interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, speakers, etc., a storage unit 18, such as a disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0069] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as method image generation.

[0070] In some embodiments, the method picture generation may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via a read-only memory (ROM) 12 and / or a communication unit 19. When the computer program is loaded into a random access memory (RAM) 13 and executed by the processor 11, one or more steps of the method picture generation described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the method picture generation in any other suitable manner (e.g., by means of firmware).

[0071] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0072] The computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor, the functions / operations specified in the flow charts and / or block diagrams are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0073] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0074] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0075] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0076] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0077] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the image generation method provided in any embodiment of the present application.

[0078] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present application, and the programming language includes an object-oriented programming language, such as Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as "C" language or similar programming language. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0079] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for generating an image, characterized in that: include: A first initial image is obtained by a real camera; wherein the real camera includes a differentiable lens encoder in a pre-trained image generation model; the image generation model also includes a decoder; the image generation model is pre-trained end-to-end by combining a deep optical tolerance with a point spread loss function and a spot loss function; The first initial picture is decoded by the decoder to obtain a first target picture.

2. The method according to claim 1, characterized in that The training method of the image generation model includes: Get the training image set; Randomly sampling the deep optical tolerance during ray tracing of the differentiable lens encoder by a tolerance sampler; Transforming the spatial position and curvature of the differentiable lens encoder based on the depth optical tolerance; Obtaining a point spread function atlas and a light spot array atlas corresponding to the training picture set through the differentiable lens encoder based on the training picture set and a ray tracing method; Determine a simulation picture set based on the point spread function atlas and the training picture set; Decoding the simulated picture set by the decoder to obtain a second target picture set; Determine a target loss value based on the training picture set, the second target picture set, the point spread function atlas, and the light spot column atlas; The image generation model is trained based on the loss value until a training stop condition is met, thereby obtaining a trained image generation model.

3. The method according to claim 2, characterized in that in, The depth optical tolerance is a tolerance that obeys Gaussian distribution; the depth optical tolerance includes tilt error, eccentricity error, center thickness error and curvature radius error.

4. The method according to claim 2, characterized in that: in, The point spread function atlas includes a plurality of point spread function atlases; the training picture set includes a plurality of training pictures; the simulation picture set includes a plurality of simulation pictures; Determining a simulation picture set based on the point spread function atlas and the training picture set includes: For each training image, based on a point spread function graph corresponding to the training image, perform a spatially variable convolution operation on the training image to obtain a second initial image; Gaussian noise is applied to the second initial image to obtain a simulated image corresponding to the training image.

5. The method according to claim 4, characterized in that The image generation model is trained based on the loss value until a training stop condition is met to obtain a trained image generation model, including: Determine a picture loss value based on the training picture set and the second target picture set; By means of the point spread loss function, a point spread loss value is determined based on the similarity between each point spread function graph in the point spread function graph set and a preset point spread function graph; wherein the preset point spread function graph corresponding to each point spread function graph is the same; Determine the spot loss value based on the spot array atlas by using the spot loss function; The image generation model is trained based on the image loss value, the point spread loss value, and the spot loss value until a training stop condition is met, thereby obtaining a trained image generation model.

6. The method according to claim 5, characterized in that in, The light spot array atlas set includes a plurality of light spot arrays; each light spot array includes a plurality of grids; each grid includes a light spot; and determining a light spot loss value based on the light spot array atlas set by using the light spot loss function includes: For each light spot array diagram, the average light spot size of the plurality of grids is determined by the light spot loss function as the loss value of the light spot array diagram.

7. A picture generating device, characterized in that: include: A first initial image acquisition module, used to obtain a first initial image through a real camera; wherein the real camera includes a differentiable lens encoder in a pre-trained image generation model; the image generation model also includes a decoder; the image generation model is pre-trained end-to-end through deep optical tolerance combined with a point spread loss function and a spot loss function; A decoding module is used to decode the first initial picture through the decoder to obtain a first target picture.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image generation method as described in any one of claims 1-6.

9. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to execute the image generation method according to any one of claims 1 to 6 when executed by a computer processor.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the picture generation method according to any one of claims 1 to 6.

Images