An imaging optimization method and manufacturing method for a camera module

Abstract

This application discloses an imaging optimization method for a camera module, comprising: capturing an image of an object through an optical lens to obtain an image of the object captured by the optical lens; establishing an optical system response model of the optical lens based on the image of the object captured by the optical lens; adding a difference term containing optical system aberrations to the optical system response model of the optical lens to obtain an aberration-containing model input; training a deep learning model based on the aberration-containing model input to obtain an optical system response difference model; and optimizing the image captured by the camera module using the optical system response difference model. This method can reduce or even eliminate the difference term that generates optical aberrations in the imaging of the camera module, thereby optimizing the imaging of the camera module.

Description

Technical Field

[0001] This application relates to the fields of image testing, computational optics and deep learning, and in particular to an imaging optimization method and manufacturing method for a camera module. Background Technology

[0002] With the rapid development of technology and the fast pace of smartphone upgrades, users have increasingly higher demands for the image quality of mobile phone camera modules. Improving the resolution of camera module images has become a key competitive factor for manufacturers. Currently, the main way to improve the pixel quality of mobile phone camera modules is to upgrade the components, such as increasing the number of lens elements in the optical lens, increasing the image sensor size, increasing the pixel area of ​​the image sensor, and adding a stabilization motor. These upgrade solutions indicate that the most direct way to improve the image quality of a camera module is to upgrade its hardware.

[0003] To improve resolution, optical lenses often increase the number of optical elements, thus increasing the overall height and width of the lens. Increasing the size of the image sensor leads to a larger camera module. Adding an image stabilization motor can counteract blur caused by user hand shake, but it increases the size and cost of the camera module. Upgrades to components within the camera module often result in increased size and cost, but smartphones consistently trend towards thinner and lighter designs, leaving limited design space for the camera module. Mainstream camera module solutions in smartphones require the smallest possible size, and due to these size constraints, it's difficult to overcome the limitations of smartphone size when upgrading components.

[0004] With the development of semiconductor technology, especially the improvement of CPU integration performance, the computing power of processor chips in mobile phones, tablets, and computers has become very powerful. Besides handling daily operating system processing, these chips still have surplus computing power for other tasks. In current technology, processor chips can also handle image fusion and rendering. In recent years, with the popularization of deep learning and the development of optical imaging technology, optical imaging has gradually evolved from traditional color imaging into the era of computational optics. Computational optics requires training deep learning models, which in turn require large datasets to optimize network structure and model parameters. Therefore, a testing method that can generate large amounts of data is needed to provide training data for deep learning models. Furthermore, the closer the dataset used is to those generated in actual manufacturing or daily life, the better the image quality will be. If there are factors in actual manufacturing or daily life that cause directional image degradation, and this degradation exhibits a certain regularity, the trained deep learning model can often compensate for it effectively.

[0005] In existing technologies, deep learning models can generally adjust the pixels, brightness, and color of images acquired by sensors to output clearer images or meet specific user needs. Deep learning models can also add details or compensation to the image, such as texture detail enhancement and blur correction. After deep training, deep learning models can optimize low-quality images into high-quality ones. Furthermore, deep learning models possess strong capabilities in other areas, including the ability to acquire and optimize image feature information and improve core performance indicators of imaging quality (such as spatial resolution, temporal resolution, and sensitivity). Therefore, computational optics models trained through deep learning can improve the imaging of camera modules solely through software without altering the hardware. Summary of the Invention

[0006] One objective of this application is to provide an imaging optimization method for a camera module to improve the imaging performance of the camera module.

[0007] One objective of this application is to provide an imaging optimization method for a camera module, which involves deep learning model training and imaging optimization based on factors that cause image orientation degradation in actual manufacturing or use.

[0008] One objective of this application is to provide an imaging optimization method that utilizes deep learning models to train models based on differentiated data obtained during manufacturing or use.

[0009] One objective of this application is to provide an imaging optimization method using a deep learning model, which collects image data closely related to the research and development and production of actual camera modules, and improves the image resolution by discovering and revealing the factors of regular image degradation and training the deep learning model with the data.

[0010] One objective of this application is to provide an imaging optimization method for a camera module, which uses factors that cause image degradation in the actual manufacturing process of the camera module to train a deep learning model and form a differentiated computational optics optimization scheme to improve the image compensation capability during computational optics.

[0011] One objective of this application is to provide a method for manufacturing a camera module, which optimizes a deep learning model using images captured by the camera module, integrates the deep model into a processor module, and provides a complete computational optics method for the camera module.

[0012] Further embodiments and features are set forth in part in the following description, and will be understood by those skilled in the art upon review of the specification or through practice of the disclosed subject matter. Further understanding of the features and advantages of this disclosure may be achieved by referring to the remainder of the specification and drawings, which form part of this application. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a method according to an embodiment of this application;

[0014] Figure 2A This is a schematic diagram of a test system according to an embodiment of this application;

[0015] Figure 2B This is a schematic diagram of a test system according to another embodiment of this application;

[0016] Figure 3A This is a schematic diagram of dot matrix plate imaging according to an embodiment of this application;

[0017] Figure 3B This is a schematic diagram of dot matrix plate imaging according to another embodiment of this application;

[0018] Figure 3C This is a schematic diagram of dot matrix plate imaging according to another embodiment of this application;

[0019] Figure 4 This is a schematic diagram illustrating the decomposition of a color image according to an embodiment of this application;

[0020] Figure 5 This is a schematic diagram illustrating the modulation processing of a standard image according to an embodiment of this application;

[0021] Figure 6 This is a flowchart of a method according to another embodiment of this application;

[0022] Figure 7 A flowchart of a manufacturing method according to another embodiment of this application;

[0023] Figure 8 A flowchart of a manufacturing method according to another embodiment of this application; Detailed Implementation

[0024] The present application will be further described below with reference to specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0025] The term "comprising" is open-ended. As used in the appended claims, it does not exclude additional structures or steps.

[0026] In the description of this application, it should be noted that the directional terms such as "center", "lateral", "longitudinal", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", and "counterclockwise" indicate the orientation and positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. They should not be construed as limiting the specific protection scope of this application.

[0027] It should be noted that the terms "first," "second," etc., in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0028] The terms “comprising” and “having”, and any variations thereof, in the specification and claims of this application are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0029] It should be noted that, as used in this application, the terms “basically,” “approximately,” and similar terms are used to indicate approximation rather than degree, and are intended to describe inherent deviations in measured or calculated values ​​that would be recognized by a person skilled in the art.

[0030] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection, a contact connection, or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0031] "Configured as" refers to various units, circuits, or other components that can be described or stated as being "configured as" to perform one or more tasks. In such a context, "configured as" is used to imply a structure by indicating that the unit / circuit / component includes a structure (e.g., a circuit) that performs this one or more tasks during operation. Furthermore, "configured as" can include a general structure (e.g., a general-purpose circuit) manipulated by software and / or firmware to operate in a manner capable of performing one or more tasks to be solved. "Configured as" can also include adjusting a manufacturing process (e.g., a semiconductor fabrication facility) to manufacture a device (e.g., an integrated circuit) suitable for implementing or performing one or more tasks.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the specification and appended claims, the singular forms “a,” “an,” and “the” are intended to also cover the plural forms unless the context otherwise expressly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the items listed in connection with the description. It will also be understood that the terms “comprising” and / or “including” as used in this specification specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0033] As used herein, depending on the context, the term "if" can be interpreted as meaning "when..." or "in response to determination" or "in response to detection". Similarly, depending on the context, the phrase "if it is determined..." or "if [the stated condition or event] is detected" can be interpreted as meaning "when it is determined..." or "in response to determination..." or "when [the stated condition or event] is detected" or "in response to detection".

[0034] Description of an exemplary imaging optimization method for a camera module

[0035] Figures 1 to 6 This application illustrates the method for improving the imaging optimization of a camera module.

[0036] In existing technologies, camera modules typically include components such as lenses, image sensors, lens mounts, and color filters. These components undergo specific manufacturing processes, such as clamping, attaching, assembling, and baking, to ultimately assemble the camera module. These components often experience directional deformation due to machine or process factors. For example, existing technologies exhibit warping of image sensors during attachment and baking, resulting in poor image quality due to warping. Existing technologies also exhibit deformation of circuit boards after pressure and baking, leading to poor image plane tilt in optical systems. Furthermore, existing technologies also exhibit deformation of the optical lens after assembly, pressure, and baking, resulting in poor image quality due to astigmatism. Meanwhile, as the number of components increases, the adverse factors of multiple components on the optical system of the entire module will eventually be superimposed after the camera module is assembled in the preset position. This will cause the imaging from the optical lens to the imaging of the camera module, which will be degraded. Conventional calibration methods will be difficult to compensate for, resulting in a decrease in the imaging performance of the camera module or an increase in the product defect rate.

[0037] This application provides an approach to optimizing the imaging of a camera module. Optical lenses are generally designed according to the diffraction limit, and their performance is often pushed to the limit during the design process. After assembling the optical lenses into an optical lens, manufacturers are limited by their manufacturing capabilities, making it difficult for the lens surface shape to match the design values. Consequently, the dimensions, specifications, and performance of the lenses often fail to meet the design specifications. Therefore, the performance of the optical lens will be slightly lower than that of the optical lens model in the software design. Furthermore, because the lens assembly conditions are simple—merely assembling the lenses together—the contact stress between them is relatively low. Therefore, the performance of the optical lens also decreases slightly compared to the design model, with some parameters, such as SFR, showing a 2%-3% decrease. The process of assembling optical lenses, image sensors, color filter holders, and actuators into a camera module involves placing more components in pre-defined positions. The assembly conditions are more complex, typically involving multiple baking processes and stress bonding. Furthermore, due to significant differences in component materials and shapes, the contact stress between components is more complex. Ultimately, the performance of the camera module deteriorates significantly compared to the design model, with some parameters, such as SFR (Surface Reflectance), showing a 10%-20% decrease. Clearly, the optical lens itself better represents the optimal imaging performance of the camera module than simply assembling it. However, current camera module manufacturing processes inevitably lead to a substantial decrease in imaging performance compared to the design value. Therefore, optimizing the imaging performance of the camera module to approach or even exceed the design value is a highly valuable task, as it can significantly improve the image quality of each module and enhance the photography experience.

[0038] Furthermore, based on years of production experience from camera module manufacturers, the performance of a batch of camera modules often follows a normal distribution. The performance difference between this batch and its theoretically designed optical performance also conforms to a normal distribution. Under constant production conditions, the pressure bonding, multiple baking cycles, stress bonding, and other factors affecting the components that degrade imaging during camera module production also follow a normal distribution. The base region dominates this distribution, representing the primary component responsible for directional deformation caused by machine or process factors. With the current large-scale manufacturing of camera modules, the use of multiple molds and batch shipments has become commonplace. This leads to more concentrated directional variations in component materials and during manufacturing on specific production lines. By collecting, analyzing, and modeling these directionally detectable differences through big data collection, and then using deep learning models for learning and correction, the imaging performance of camera modules can be effectively optimized and improved.

[0039] Therefore, one approach of the imaging optimization method for a camera module provided in this application is to perform performance testing on optical lenses. After assembling the optical lenses into a camera module, performance testing is conducted. By analyzing and recording the field of view, aberration characteristics, and aberration quantification values ​​where optical performance deteriorates after a single lens is assembled into the camera module, and then continuously performing deep learning through big data, the performance degradation factors generated when assembling optical lenses into a camera module can be completely eliminated. Obviously, since the performance degradation factors are normally distributed, the deep learning model can identify the main influencing factors and make subsequent modifications or image optimizations, thus improving the main defects in the product and achieving a high product yield.

[0040] In existing technologies, the point spread function (PSF) is used in imaging systems to respond to point light sources or point objects. The PSF focuses on the impulse response of the optical system and is the spatial domain expression of the optical transfer function of the imaging system, reflecting the aberrations of the optical system. In other words, the PSF can obtain the relationship between points on the object side and points on the image side, thus reflecting the actual light transmission process of the entire optical system and obtaining the aberrations of the optical system from point to surface. However, in the field of camera module manufacturing, only important fields of view are often controlled, such as 0, 0.3, 0.5, 0.7, and 0.8 fields of view. Therefore, it is only necessary to measure the important fields of view, and selecting important test points can represent the measurement of the entire camera module.

[0041] Reference Appendix Figure 1 As shown, this application proposes an imaging optimization method for a camera module, which includes the following steps:

[0042] S1: Take a picture of the object through the optical lens and obtain an image of the object taken by the optical lens;

[0043] S2: Establish an optical system response model of the optical lens based on the image of the object captured by the optical lens;

[0044] S3: Based on the optical system response model of the optical lens, add a difference term including optical system aberrations to obtain the model input terminal containing aberrations;

[0045] S4: The deep learning model is trained based on the aberration-containing model input to obtain the optical system response difference model;

[0046] S5: Optimize the images captured by the camera module using the optical system response difference model.

[0047] More specifically, according to a first aspect of the embodiments of this disclosure, reference is made to the appendix... Figure 1 As shown in the flowchart, this application provides an imaging optimization method for a camera module, characterized by comprising the following steps:

[0048] S1: Take a picture of the object through the optical lens and obtain an image of the object taken by the optical lens;

[0049] In step S1, it is preferable to use a single optical lens for shooting. This reduces the risk of shape variations in the optical lens caused by manufacturing processes during the assembly of the camera module, which can negatively impact the imaging performance of the camera module. In another embodiment, it can also refer to shooting through the optical lens while the camera module is in its assembled state. Therefore, step S1 can refer to the process of shooting through an optical lens in a broad sense. This step is primarily aimed at acquiring images captured through the optical lens to obtain the foundational image information for subsequent image optimization.

[0050] S2: Establish an optical system response model of the optical lens based on the image of the object captured by the optical lens;

[0051] In step S2, after the object is photographed through the optical lens, an optical system response model of the optical lens is established based on the acquired imaging information and object information. To understand the relationship between the photographed object and the image, it is necessary to know how the optical transfer function (OPF) changes from object to image. Step S2 establishes the OPF of the object after passing through the imaging system. After obtaining the OPF, the image output by the object in response to the optical system can be obtained. With the OPF response model containing the OPF, it is not necessary to photograph again; the theoretically close image to the photographed image can be output by superimposing the OPF response model on different objects (image formats). Step S2 primarily aims to acquire the image captured through the optical lens to obtain the basic image information for subsequent image optimization. Therefore, step S2 can refer to acquiring the OPF in a broad sense; however, sometimes, to adapt to more situations, acquiring the OPF response model is more suitable for cases where the OPF cannot fully reflect the object-to-image transition.

[0052] S3. Based on the optical system response model of the optical lens, add a difference term including optical system aberrations to obtain the model input terminal containing aberrations;

[0053] In a preferred embodiment of this application, the preceding steps S1 and S2 establish an optical system response model for a single lens. However, as described above, this model lacks factors causing optical aberrations, thus failing to reflect the optical performance of a batch of camera modules. For example, the optical system response model after steps S1 to S2 lacks aberrations caused by actual manufacturing processes, errors due to limitations of the optical system itself during actual use, and aberrations caused by batch production issues. These missing optical aberrations can be introduced into the optical system response model as difference terms (parametric equations), enabling the model after step S3 to simulate the factors causing optical aberrations that should exist during the actual production and final shooting process of the camera module. This allows camera module manufacturers to utilize big data from camera modules on a digital platform and output inputs that simulate the actual production and final shooting process of the camera module. In another approach, targeted optical aberrations can be superimposed on the optical system response model. For example, camera module manufacturers often need to address the issue of chip warping caused by high-temperature baking of large-image-size chips. Chip warping often leads to field curvature in actual imaging. This application allows for the identification of camera modules with chip field curvature issues in step S3. The distribution and properties of the field curvature in the optical system of these problematic modules can be analyzed. By introducing targeted difference terms for field-curvature optical aberrations, the optical system response model can more closely approximate the actual imaging of the camera module. Subsequent deep learning models can then learn and improve the imaging field curvature of the optical system, thereby specifically improving the imaging capabilities of the actual camera module.

[0054] In other aspects, this method of superimposing difference terms on the basic optical system response model can adapt to more complex situations. For example, it can simulate the normal distribution of optical performance peaks according to the aforementioned gradient, selecting different gradient optical performance as difference terms to simulate the actual normal distribution in real-world camera module products. It can also introduce difference terms to characterize different optical aberrations, such as field curvature, peak performance, and on-axis defects, to more closely reflect the various defects that occur in camera modules in real-world situations. This allows the optical system response model to simulate more camera module scenarios, encompassing more complex situations in actual production and truly analyzing the performance of camera modules in real-world production processes using big data.

[0055] S4. The deep learning model is trained based on the aberration-containing model input to obtain the optical system response difference model;

[0056] In step S4, since the preceding steps have yielded a simulation of the camera module's imaging and established an optical system response model that includes difference terms related to optical system aberrations, and because it incorporates various aberrations present during actual camera module imaging, this optical system response model is now capable of outputting simulated production batch camera module information based on the input object information. Therefore, the model input can be considered close to the image captured by the actual camera module. Furthermore, since the digitized optical system response model allows for the output of simulated images that closely resemble those captured by the actual production batch camera module without actually capturing images with the production batch camera module, the model input (the image output by the digitized optical system response model according to requirements) is sufficient to serve as the image information captured by the actual camera module required by the deep learning model. This approach simplifies the data generation process required for deep learning models. In this application, the optical system response model, which incorporates differences in optical system aberrations, can completely replace images captured by camera modules in actual production batches. By simply using the image of the input object and evolving the model, images simulating those captured by camera modules in actual production batches can be obtained. This method significantly simplifies the data collection work required for deep learning models. Without this approach, each camera module would need to be photographed according to different requirements, and camera modules representing different gradients would need to be selected from different batches. Furthermore, each camera module would need to be photographed in different testing environments to obtain the input data required by the deep learning model.

[0057] Based on the data provided by the model's input, the deep learning model can optimize images according to its network structure, ultimately obtaining the optical system response difference model for that production batch of camera modules. The optical system response difference model records the locations and optimization methods of different regions in the images before and after optimization, and also includes the optimization methods for different types of optical aberrations in the images. After optimization, the required optimization locations, optical aberration types, and optimization methods are recorded and updated to form the optical system response difference model.

[0058] S5: Optimize the images captured by the camera module using the optical system response difference model;

[0059] In step S5, since the optical system response model has been completed in the previous steps, meaning the model can handle various defects of the camera modules in the actual production batch, the images actually captured by the camera modules in the corresponding production batch can be optimized by inputting them into the optical system response difference model.

[0060] It is worth mentioning that, in the step of adding a difference term including optical system aberrations to obtain the model input terminal containing aberrations based on the optical system response model of the aforementioned optical lens: the difference term including optical aberrations includes factors that cause wavefront aberrations in the optical system from the design, production, manufacturing, and use of the camera module. As mentioned above, the production process of the camera module can lead to optical aberrations in the optical system; the design of the camera module's optical system can also cause optical aberrations during actual shooting by the camera module; and the camera module can also generate optical aberrations during use, leading to actual shooting by the camera module.

[0061] The S1 step of capturing an image of an object through an optical lens includes: capturing an image of a dot matrix target through the optical lens and acquiring the image of the dot matrix target captured by the optical lens. The foregoing discussion proposed obtaining the point spread function (PSF) of the optical lens, which is used in the imaging system's response to a point source or point object. The PSF focuses the impulse response of the optical system, reflecting the system's response to the point source, and thus allows for the acquisition of wavefront aberrations, etc., of the optical system. In other words, in this application, the PSF of the optical lens is used as the transfer function of the optical lens to obtain the response model of the optical system.

[0062] Step S2 – Establishing the optical system response model of the optical lens based on the imaging of the object captured by the optical lens – includes: establishing the optical system response of the optical lens to a point source based on the imaging of a dot matrix target captured by the optical lens. The foregoing mentions obtaining the point spread function (PSF) of the optical lens for use in the imaging system's response to a point source or point object. The PSF focuses the impulse response of the optical system, reflecting the system's response to the point source, thus allowing the acquisition of wavefront aberrations, etc. In other words, in this application, the PSF of the optical lens is used as the transfer function of the optical lens to obtain the response model of the optical system.

[0063] The step of adding a difference term containing optical system aberrations to the optical system response model of the optical lens described in S3 to obtain the aberration-containing model input includes:

[0064] S3.1: Assemble the optical lenses into a camera module according to the specified manufacturing process;

[0065] S3.2: Capture the dot matrix marker using the camera module to obtain an image of the dot matrix marker captured by the camera module;

[0066] S3.3: Establish the response of the camera module's optical system to the point source based on the dot matrix target image captured by the camera module;

[0067] S3.4: The difference between the optical system response of the camera module to a point source and the optical system response of the optical lens to a point source is processed into a difference item from the optical lens to the camera module;

[0068] S3.5: The optical system response model of the optical lens is used as a basis to add a difference term from the optical lens to the camera module;

[0069] The foregoing description explains that during the assembly of optical lenses into a camera module, the manufacturing process can affect the shape of the optical lenses, leading to physical variations and deformation of the image sensor. Ultimately, this results in a deterioration in the imaging performance of the camera module. Differences in optical aberrations caused by manufacturing processes are considered, according to manufacturer practice, the biggest factor affecting image formation in camera modules. This application compensates for these manufacturing factors in the camera module. Supplementary steps S3.1 to S3.5 include photographing a point-source target image of the camera module and obtaining the response of the optical system in the camera module state after assembly according to the specified process.

[0070] In the foregoing, obtaining the point spread function can be considered a good way to understand the response of the optical system to a point source. In a preferred embodiment of this application, after obtaining the point source response of the camera module optical system, it is then differentiated from the point source response of the optical lens optical system to obtain the difference terms in the response to the point source from the optical lens to the camera module assembled according to the specified process. The difference terms from the optical lens to the camera module indicate the changes in the optical performance of the optical lens itself after assembly into the camera module, specifically the differences in the response of the optical system to the point source (e.g., the location, type, and degree of optical aberrations mentioned above). The degradation factors of the optical system are identified as difference terms, so that targeted degradation factors appearing in the camera module manufacturing process can be corrected and optimized through subsequent deep learning models.

[0071] In the S4-deep learning model training step to obtain the optical system response difference model based on the aberration-containing model input, it also includes: S4.1: taking the optical system response of the optical lens to the point source as the input and the optical system response of the camera module to the point source as the output, and establishing a point response difference model from the optical lens to the optical system assembled into the camera module based on the difference of the same point source response from the input to the output.

[0072] In prior art, deep learning models are continuously trained by optimizing the input image into a target image. As mentioned earlier, optical aberrations caused by manufacturing processes of the camera module are considered a major problem causing image fragmentation in the camera module assembly. Step S4.1 uses the response of the optical system of the lens to a point source as the input and the response of the camera module's optical system to a point source as the output. Establishing a point response difference model from the optical lens to the optical system assembled into the camera module based on the differences in the responses to the same point source from the input to the output can be considered another embodiment of this application. Steps S3.1 to S3.5 above superimpose the difference terms of optical aberrations caused during the camera module manufacturing process into the model, providing the data needed for input training of the deep learning model. However, as mentioned earlier, if the image of the camera module can be optimized to near the imaging quality of a single optical lens, the imaging performance of the camera module can be greatly improved due to the reduction of optical aberrations caused by manufacturing processes. Therefore, step S4.1 can continuously train the input image (image in the state of being assembled into a camera module) with the target image (image in the state of being a single lens) to obtain an image that can eliminate the factors that affect the imaging of the camera module manufacturing process.

[0073] In step S5, which optimizes the image captured by the camera module using the optical system response difference model, the following step is also included: optimizing the image captured by the camera module using the point response difference model of the optical system from the optical lens to the optical system assembled into the camera module.

[0074] Since the optical system point response model is considered the optimal solution for acquiring the wavefront of the optical system, and because it can output images close to those actually captured by camera modules in production batches after inputting the image information of the object into the optical system point response model, the optical system point response difference model is obtained by optimizing the original optical system point response difference model after the deep learning model learns the difference between the optical system point response model and the target optical system point response model. This results in a point response difference model of the optical system from the optical lens to the assembled camera module. As described above, the optical system from the optical lens to the assembled camera module is affected by the manufacturing process, and the optical system of the camera module is degraded due to manufacturing process factors. Therefore, obtaining the point response difference model of the optical system from the optical lens to the assembled camera module can solve most of the image degradation effects caused by the manufacturing process.

[0075] In another embodiment of this application, the difference term including optical aberrations includes aberrations produced by the optical system of the camera module in near-focus and far-focus images due to the optical system design. The near-focus and far-focus images are caused by the deflection of rays at infinity and near distance due to the optical design. After the optical system is determined and the camera module is assembled through a specified process, the difference between the near-focus and far-focus images becomes largely directional, and therefore can be corrected directionally using a deep learning model. Unlike degradation factors caused by improving the camera module manufacturing process, improving the aberrations produced by the optical system of the camera module in near-focus and far-focus images largely stems from the optical design.

[0076] In another embodiment of this application, the difference including optical aberrations includes aberrations in the optical system of the camera module caused by process factors during the camera module assembly process. As described in the foregoing sections, the camera module manufacturing process can cause further optical aberrations due to process effects. Eliminating aberrations caused by these process factors can greatly improve the imaging performance of the camera module.

[0077] Therefore, this application provides a method to obtain the point source response (point spread function) of the optical system in lens mode by testing the lens, and the point source response (point spread function) of the optical system in camera module mode can also be obtained by testing the camera module. This application requires measuring the point source response of the optical system in lens mode and the optical system in module mode separately, so that the test data includes information on which point source responses are ultimately affected by factors causing image degradation during the lens assembly to camera module manufacturing process, and the differences between the same point sources in the optical systems of the optical lens and the camera module, in order to specifically compensate for the imaging issues of the camera module.

[0078] In existing technologies, the extraction of the point spread function (PSF) requires imaging of the point source through an optical system. However, obtaining image information of a point light source in real-world scenarios is difficult. Typically, in laboratories, PSF extraction involves generating a bitmap using a computer and then photographing the printed bitmap to obtain the PSF intensity map. This method is simple and convenient, requiring no testing environment. However, the imaging object is not a point light source but a black-and-white bitmap, introducing factors such as the uniformity of light source brightness and aberrations present during printing. Therefore, the obtained PSF has low accuracy and limited practical value; this method cannot obtain highly accurate PSF information.

[0079] To address at least one of the aforementioned problems, this application designs a high-precision point spread function (PSF) extraction target based on a uniform light source. This target has a dot matrix pattern and can therefore also be called a dot matrix target. The dot matrix pattern includes dots corresponding to the pixel size of the image sensor. This dot matrix target is designed based on the lens design parameters in the camera module and the required magnification during lens design. The dot matrix pattern can be manufactured with high precision according to different design requirements and shooting needs. In this application, the test point size of the dot matrix pattern is equivalent to 1-2 unit pixels in the image sensor, thereby enabling pixel-level PSF function measurement. Therefore, it meets the accuracy requirements for calculating optical PSF. Generally, the dot size of the dot matrix target pattern in this application needs to be greater than the diffraction limit in the optical system, and the smaller the size of the dot matrix pattern, the higher the measurement accuracy.

[0080] The dot matrix marker includes a light source and a dot pattern. Light emitted from the light source is patterned after passing through the dot pattern to propagate uniformly outwards. The light source is suitable for adjusting color temperature, changing wavelength, and altering brightness. In this application, the light source is preferably a laser source to ensure high collimation capability. Overly dispersed light from the light source, after passing through the lens and being received by the photosensitive chip, will form a blurred spot, thus affecting the calculation of the dot spread function. Generally, the better the collimation of the light source, the smaller the spot size, and the more accurate the brightness and position information output by the photosensitive chip after receiving the spot signal.

[0081] The point source response, or point response, or point object response mentioned above can be calculated using the following formula: u i (x, y) = u g (x, y) × h (x, y) + n (x, y), here, u i (x, y) is the image plane of the point source, u g (x, y) is the object plane of the point source, h(x, y) is the point spread function (also known as the PSF function), and n(x, y) is the current noise. In other words, this application shows that by using the object plane of the point source, the point spread function of the optical system, and the current noise, the image of the point source and various aberrations during imaging can be obtained.

[0082] As described above regarding the technical aspects of the dot matrix marker, the dot matrix marker in this application includes a light source and a dot pattern. The light emitted by the light source is patterned after passing through the dot pattern to propagate uniformly outward. The light source is suitable for adjusting its color temperature, changing its wavelength, and altering its brightness. Because the light source can have its color temperature adjusted, its wavelength changed, and its brightness altered, it can provide different testing environments to meet various testing needs.

[0083] The step of capturing an image of a dot matrix target using an optical lens can be broken down into the following actions: The distance between the dot matrix target and the optical lens along the optical axis is changed; the optical lens is then focused; images of the dot matrix target are captured using the optical lens at different distances relative to the optical lens along the optical axis; and a mapping between the focusing distance of the optical lens and the image is established based on the distance between the optical lens and the dot matrix target during image capture. This method aims to establish a relationship between the captured dot matrix image and the focusing distance, thereby satisfying the near-to-far focus tests required for general camera module factory calibration. Furthermore, it increases the sample data of test images, thus increasing the amount of data tested.

[0084] The aforementioned change in the distance between the dot matrix marker and the optical lens along the optical axis refers to moving the dot matrix marker along the optical axis and capturing images at different distances to obtain images of the dot matrix marker at near and far focal lengths. This situation refers to the fact that in actual shooting, users often shoot objects at different distances, such as distant mountains or close-up portraits. Different focusing distances will produce images at different out-of-focus positions. In the design of optical systems, parallel light rays at infinity are often used as input light, and the optimization of optical systems is based on optimizing parallel light rays at infinity as a prerequisite. In actual shooting, users often shoot close-up scenes, or even close-up selfies. Therefore, in actual production, close-up shooting often accounts for the majority of shots. Improving close-up shooting can improve the shooting effect when shooting at normal distances. Therefore, this solution will test the dot matrix marker at close distances to provide images captured at close distances. One of the important objectives of this application is to acquire close-up images and optimize them through post-processing.

[0085] However, while most user-captured scenes are considered close-up, the optical performance of an optical system often differs significantly between close and long distances from the initial design stage. This inherent difference in lens design is often further degraded during the actual production of the camera module. In actual imaging, this difference stems from the design of the optical lens itself; that is, the point spread function of the optical lens causes the difference in imaging performance between close and long distances. Furthermore, optical lenses can have two different sets of point spread functions for close and long distances. These two sets of point spread functions can lead to significant differences in the images captured by the lens at close and long distances. Therefore, the difference in optical performance between close and long distances can result in inconsistent image quality due to lens factors. For example, a long-distance image might be very clear, while a close-up image might be of poor quality. Generally, once the lens's optical design is determined, this difference in optical performance between close and long distances is predetermined and exhibits a directional degradation phenomenon. This difference can be eliminated by using large datasets to train deep learning models.

[0086] This application provides a dot matrix target that can test any defocus position, thereby enabling the dot matrix target to be adjusted to different distances, such as within 1.5 meters and beyond 1.5 meters. By superimposing a teleconverter or using a working axis to move the dot matrix target, the relative optical distance between the target and the imaging chip can be adjusted from near to infinity, thus adapting to different shooting needs.

[0087] The process of capturing an image of a dot matrix target using a camera module can be further refined to include the following steps: changing the distance between the dot matrix target and the camera module along the optical axis of the camera module; focusing the camera module; capturing images of the dot matrix target at different distances relative to the camera module along the optical axis of the camera module; and establishing a mapping between the focusing distance of the camera module and the image based on the distance between the camera module and the dot matrix target when capturing the image. This method aims to establish a relationship between the captured dot matrix image and the focusing distance, thereby satisfying the near-far focus tests required for general camera module factory calibration. It also aims to increase the sample data of test images, thus increasing the amount of data tested. More importantly, unlike the focusing of the optical lens mentioned above, the focusing of the camera module often represents the state after the motor has worked and driven its stroke. Therefore, having the camera module take pictures after the motor has worked can obtain the imaging performance of the camera module under the influence of the motor. The captured data can also include the influence of the motor as a factor that degrades image quality. For example, if the motor has poor orientation tilt, the poor orientation tilt of the motor can also be included in the image plane tilt caused by the poor orientation tilt of the motor.

[0088] Similar to the aforementioned adjustment of focus by changing the distance between the dot matrix marker and the optical lens along the optical axis, in order to improve the near-focus images captured by the camera module, this application adjusts the focus state of the camera module by sending various electrical signals to the motor. This allows the camera module to capture near-focus and far-focus images at different distances, thus enabling the acquisition of near-focus and far-focus images captured by the camera module at both near and far-focus positions, providing a basis for subsequent image correction.

[0089] This refers to moving the dot matrix marker along the optical axis and capturing images at different distances to obtain images of the dot matrix marker at near and far focal lengths. Therefore, it can include near and far focal lengths, capturing images at close or far distances, respectively, when the dot matrix marker is at its near and far focal lengths. This situation refers to the fact that in actual shooting, users often photograph objects at different distances, such as distant mountains or close-up portraits. Different focal distances produce images at different out-of-focus positions. In the design of optical systems, parallel rays from infinity are often used as input light, and the optimization of optical systems is based on optimizing parallel rays from infinity as a prerequisite. In actual shooting, users often photograph close-up objects, even close-up selfies; therefore, close-up shooting often accounts for the majority of shots in actual production. Improving close-up shooting can improve the shooting effect when shooting at normal distances. Therefore, this solution will test the dot matrix marker at close distances to provide images captured at close distances.

[0090] Most shooting scenarios involve close-up shots. From the initial design of the optical system, its optical performance often differs significantly between close-up and long-distance shots. These inherent differences, stemming from lens design, tend to worsen during the actual production of the camera module. In actual imaging, these differences originate from the design of the optical lens itself; specifically, the point spread function of the lens leads to variations in imaging performance between close and long distances. Furthermore, the change in object distance between close-up and long-distance shots alters the image height of the corresponding object, resulting in differences in object size in the optical system's imaging. This difference is amplified by the pixels of the CMOS sensor within the camera module, as the fixed pixel size further amplifies these discrepancies. Therefore, in actual use, camera modules exhibit significant differences in image quality between close-up and long-distance shots captured by the optical lens. This difference in optical performance at close and long distances leads to inconsistent image quality due to lens limitations; for example, a long-distance image might be very clear, while a close-up image might be of poor quality. Generally, once the lens's optical design is determined, this difference in optical performance between close and long distances is predetermined and exhibits a directional degradation. This difference can be mitigated by using large datasets to train deep learning models.

[0091] This application provides a dot matrix target that can test any defocus position, thereby enabling the dot matrix target to be adjusted to different distances, such as within 1.5 meters and beyond 1.5 meters. By superimposing a teleconverter or using a working axis to move the dot matrix target, the relative optical distance between the target and the imaging chip can be adjusted from near to infinity, thus adapting to different shooting needs.

[0092] The step of establishing a point response difference model from the optical lens to the assembled camera module optical system, using the optical system's response to a point source as the input and the camera module's response to a point source as the output, may include the following steps: both the dot matrix image captured by the optical lens and the dot matrix image captured by the camera module are grayscale processed, and the images are segmented according to the grayscale value distribution to obtain pixel blocks of the dot matrix image captured by the optical lens and the dot matrix image captured by the camera module, respectively.

[0093] The process of establishing a point response difference model from the optical lens to the optical system assembled into the camera module, using the optical system's response to a point source as the input and the camera module's response to a point source as the output, and based on the differences in the responses to the same point source from the input to the output, may include the following steps:

[0094] A1.1: Extract the point source response information of each point in the input terminal, extract the intensity information map of each color channel of the single point source in the input terminal, and extract the point spread function information of each color channel of the input terminal.

[0095] A1.2: Extract the point source response information of each point in the output terminal, extract the intensity information map of each color channel of the single point source in the output terminal, and extract the point spread function information of each color channel in the output terminal.

[0096] A1.3: Establish a point response difference model from the optical lens to the optical system assembled into a camera module by combining the point spread function information of each color channel at the input end and the point spread function information of each color channel at the output end.

[0097] After capturing the target image, the embodiments of this application require processing the information contained in the image, including measuring the sharpness of the image information. In the embodiments of this application, the point response information of the optical image includes the degree of black-and-white blurring of the point pattern; therefore, the point spread function information includes a measurement of image sharpness. In subsequent steps, point spread function information of identical points on the optical lens and camera module can be extracted using this test environment, followed by differential analysis to establish a point response difference model from the optical lens to the assembled camera module optical system. This allows for large-sample deep learning to compensate for degradation factors occurring during the camera module assembly process.

[0098] Reference Appendix Figure 2A As shown, this application proposes a testing system for a dot matrix marker, including a light source 10, a dot matrix marker 20, a relay lens 30, an optical lens 40, a photosensitive chip 50, and a defocusing mechanism 60. The light emitted by the light source 10 is patterned after passing through the dot pattern of the dot matrix marker 20 to propagate uniformly outwards. The light source 10 is suitable for adjusting its color temperature, changing its wavelength, and altering its brightness. Because the light source 10 can have its color temperature adjusted, its wavelength changed, and its brightness altered, it can provide different testing environments to adapt to different testing needs.

[0099] The relay lens 30 is disposed between the optical lens 40 and the array plate 20. The relay lens 30 is used to amplify the optical path of the light from the array plate 20 to the optical lens 40, so as to extend the test distance of the test system and adapt to the changes in the mid-to-long focal length of the test distance.

[0100] The photosensitive chip 50 is disposed below the optical lens 40. The photosensitive chip 50 is used to receive light modulated by the optical lens 40, thereby enabling image capture. The defocusing mechanism 60 is disposed below the photosensitive chip 50, thereby defocusing the photosensitive chip 50 so that it is positioned at a suitable back focus position of the optical lens 40 to obtain a clear image.

[0101] The dot matrix pattern provided in this application has a dot pattern adapted to the pixel size of the photosensitive chip in the imaging system. In this embodiment, the size of a single dot pattern is exactly the diameter of two pixels of the photosensitive chip, so as to realize the calculation of 4-in-1 pixels in the photosensitive chip, thereby enabling the scaling of the image. This facilitates the synthesis of low-pixel photosensitive chips using high-pixel photosensitive chips, thereby improving the versatility of the photosensitive chip. Without changing the photosensitive chip, only changing the image synthesis algorithm can achieve the simulation of photosensitive chips of various specifications. On the other hand, by adapting to the pixel size of the photosensitive chip, it is possible to use a general-purpose photosensitive chip to meet various testing requirements.

[0102] The testing system can test the point response information of lens optics at any field of view, wavelength, and defocus position. It features high precision, strong stability, and strong compatibility. The working axis that moves the light source can also move the dot matrix specimen. This working axis can adjust the light source with a precision of 0.1µm on a plane parallel to the imaging plane. Furthermore, the working axis can also move the light source with a precision of 0.1µm along the optical axis, thus enabling high-precision movement of the light source. Driven by the working axis, the light source can move along the lens optical axis and also in a plane perpendicular to the lens optical axis. This allows for the provision of dot matrix patterns with different near and far distances relative to the imaging system, as well as dot matrix patterns with different offsets relative to the imaging system.

[0103] Reference Appendix Figure 2B As shown, this application proposes another testing system for dot matrix targets, including an object 20a, a first optical lens component 41a, a second optical lens component 42a, a photosensitive chip 50a, and a defocusing mechanism 60a. In this embodiment, the light reflected or emitted by the object 20a is received by the photosensitive chip 50 after passing through the first optical lens component 41a and the second optical lens component 42a.

[0104] In this embodiment, the purpose is to demonstrate that even with multiple lens components or separate lenses, object information captured by the optical lens can still be obtained after active calibration using the first optical lens component 41a and the second optical lens component 42a. With multiple lens components or separate lenses, there are more variables; for example, the first optical lens component 41a and the second optical lens component 42a may come from different batches, representing different physical dimensions during lens component production, resulting in different optical performances. During active calibration, the first optical lens component 41a and the second optical lens component 42a may exhibit different performances after calibration, with some batches performing well and others poorly, or showing directional performance deviations across batches. That is, batch-specific performance errors often occur in a directional manner, making them relatively easy to correct using deep learning models.

[0105] The defocusing mechanism 60 is disposed on the lower side of the photosensitive chip 50a, thereby realizing the defocusing processing of the photosensitive chip 50a, so that the photosensitive chip 50a is located at a suitable back focus position of the first optical lens component 41a and the second optical lens component 42a, so as to obtain a clear image.

[0106] Reference Appendix Figure 3AThis illustration demonstrates the imaging of a dot matrix target in this application, using a single dot as the dot matrix test area to test different test fields of view, thereby obtaining response information from point sources that can reflect the overall image. (See appendix.) Figure 3A The test fields of view are schematically divided into four areas: the central field of view 1A, the 0.3 field of view 2A, the 0.5 field of view 3A, and the 0.8 field of view 4A. Each area includes at least one test point. Point sources at the same distance from the central field of view 1B can be located at the four corners of the screen: the upper left, upper right, lower right, and lower left. This pattern continues, resulting in four test areas for each of the 0.3, 0.5, and 0.8 fields of view. This testing method allows for the measurement of important fields of view within the image.

[0107] In this application, the process of extracting point spread function (PSF) information from image information includes first obtaining a bitmap with adjusted parameters, then extracting the intensity information maps of each channel, locating test points, and finally extracting the PSF information of each point based on the test points. Generally, this can include the following formula:

[0108] According to the following formula: RImage, GImage, BImage = Extract(RGBImage)

[0109] (Point xi Point yi = Location(RImage)

[0110] PSF (Point) xi Point yi = Optain(RImage(xi, yi))

[0111] Reference Appendix Figure 4The diagram illustrates the decomposition of a color image according to its color channels. Whether the image is captured by the optical lens or the camera module, it contains RGB colors. Because different colors of light have different wavelengths, optical systems experience variations in refraction and optical path difference. These variations can lead to malfunctions in the optical system. Therefore, to further analyze the information in the image, it is necessary to decompose the image according to its color channels. In other words, this application requires taking the captured RGB image (three-channel bitmap image), RImage, GImage, and BImage as intensity maps of the R, G, and B channels, respectively. This allows for the generation of intensity maps decomposed by color channels, resulting in grayscale images of the R, G, and B colors. This allows us to extract the location of test points from the grayscale images of color R, color G, and color B, respectively, based on the location of the test points in each of the three grayscale images. Then, the formula PSF(Point...) is applied... xi Point yi The PSF data of each test point is extracted to obtain the point spread function of the point source under the three colors R, G, and B.

[0112] Reference Appendix Figure 3B This illustration shows another embodiment of the present application, in which multiple points are used as the dot matrix test area instead of the attached... Figure 3A Using a single point as the test area increases the test field of view, thus better reflecting the response information of point sources in the overall image. (See attached...) Figure 3B The test fields of view are schematically divided into four areas: the central field of view 1B, the 0.3 field of view 2B, the 0.5 field of view 3B, and the 0.8 field of view 4B. Each area includes at least four test points. The point sources at the same distance from the central field of view 1B can be the four corners of the screen: the upper left, upper right, lower right, and lower left. Similarly, the test areas for the 0.3, 0.5, and 0.8 fields of view are all four. In addition to measuring important fields of view in the screen, this test method can also achieve, for example, setting four test points on the central field of view (1), which can increase the number of sampling points and improve the accuracy. The multiple test points can also cover more pixels in the image and reduce the impact of individual endpoints.

[0113] Figure 3C This is a schematic diagram of dot matrix plate imaging according to another embodiment of this application. In another embodiment of this application, a uniformly arranged dot matrix pattern is used instead of a dot matrix plate. Figure 3AUsing a single point as the test area can greatly increase the area of ​​the test field of view, thus better reflecting the response information of point sources in the overall image. (See attached...) Figure 3C The diagram schematically divides the entire image into various test areas. Although the central field of view 1C is not shown, areas at the same distance from the central field of view can be considered as test areas of the same field of view. The 0.3 field of view 2C, 0.5 field of view 3C, and 0.8 field of view 4C each include at least one test point. Point sources at the same distance from the central field of view 1B can be the four corners of the image: upper left, upper right, lower right, and lower left. This pattern continues, resulting in four test areas for each of the 0.3, 0.5, and 0.8 field of view. This testing method allows for the measurement of important fields of view within the image. However, in this embodiment, the uniformly arranged test points will inevitably increase the test field of view, thus providing richer statistical data on point responses, but also increasing the computational load.

[0114] It can be seen that in this application, multiple points can be used as a single test area to increase the statistical number of point responses, or multiple points can be evenly arranged as a single test area to increase the statistical number of point responses. Both of these implementation methods can achieve the goal of increasing the number of test points to cover more pixels in the image and reducing the impact of individual endpoints.

[0115] In existing technologies, when users take photos with their mobile phones, the main color tone in the picture often changes according to different environments. For example, under moonlight, the picture is more likely to be filled with cool colors, while under sunlight, the picture is more likely to be filled with warm colors. Therefore, the complexity of the actual shooting environment determines that the camera module needs to adapt to different color temperatures.

[0116] To better suit actual shooting environments, the light source of the testing system in this application needs to provide various brightness and color temperature conditions. The light source of the testing system in this application is a uniform light source capable of adjusting its color temperature, illuminance, and wavelength. This allows for changes in the light source's color temperature, wavelength, and brightness, enabling the acquisition of chart images at different brightness levels, color temperatures, and wavelengths. This results in images with different color temperatures, wavelengths, and brightness levels, providing images with gradient divisions. In real-world shooting environments, the wavelength, color temperature, and brightness of light sources are not singular components. Differences in these factors often lead to variations in the captured images. For example, in cool-toned environments, the image tends to lean towards green or blue, resulting in insufficient red color information and a lack of reddish details.

[0117] In one embodiment of this application, the step of acquiring the dot matrix target image captured by the optical lens can be refined into the following actions: changing at least one of the color temperature, wavelength, and brightness of the light source; focusing the optical lens; and capturing an image of the dot matrix target image through the optical lens while changing at least one of the color temperature, wavelength, and brightness of the light source. A mapping is established between the data of the changed color temperature, wavelength, and / or brightness of the light source and the imaging data. This method aims to establish a relationship between the captured dot matrix image and the data of the color temperature, wavelength, and / or brightness of the light source, thereby satisfying the color temperature, wavelength, and brightness testing of the light source required for general camera module factory calibration. Furthermore, it also aims to increase the sample data of the test images, thereby increasing the amount of data tested.

[0118] In one embodiment of this application, the step of acquiring the dot matrix target image captured by the optical lens can be refined into the following actions: changing at least one of the color temperature, wavelength, and brightness of the light source; focusing the optical lens; capturing an image of the dot matrix target through the optical lens while changing at least one of the color temperature, wavelength, and / or brightness of the light source; and establishing a mapping between the data of the changed color temperature, wavelength, and / or brightness of the light source and the imaging data captured by the optical lens. This method is to establish a relationship between the captured dot matrix image and the data of the color temperature, wavelength, and / or brightness of the light source, thereby satisfying the color temperature, wavelength, and / or brightness testing of the light source required for general camera module factory calibration. On the other hand, it also increases the sample data of test images, thereby increasing the amount of data tested.

[0119] In one embodiment of this application, the step of acquiring the dot matrix target image captured by the camera module can be refined into the following actions: changing at least one of the color temperature, wavelength, and brightness of the light source; focusing the optical camera module; capturing an image of the dot matrix target through the camera module while changing at least one of the color temperature, wavelength, and brightness of the light source; and establishing a mapping relationship between the data of the changed color temperature, wavelength, and / or brightness of the light source and the imaging data obtained through the camera module. This method is to establish a relationship between the captured dot matrix image and the data of the color temperature, wavelength, and / or brightness of the light source, thereby satisfying the color temperature, wavelength, and brightness tests of the light source required for general camera module factory calibration. On the other hand, it also increases the sample data of test images, thereby increasing the amount of data tested.

[0120] In one embodiment of the application, a difference model of the color temperature, wavelength, and / or brightness of the light source from the optical lens to the camera module is obtained by establishing a mapping between data of changing the color temperature, wavelength, and / or brightness of the light source and imaging data through the camera module, and a mapping between data of changing the color temperature, wavelength, and / or brightness of the light source and imaging data through the optical lens.

[0121] In one embodiment of the application, the relative position of the dot matrix marker to the lens is changed according to the lens design parameters. As a relatively important optical component in the camera module, the optical lens is generally designed according to certain requirements, such as TTL, resolution, and shooting field of view. In actual lens performance control, the field of view position and resolution of the lens are often controlled. For example, a resolution of 0.8 field of view needs to meet certain performance requirements, while there are few requirements for images with a field of view of 0.9-1. Therefore, changing the relative position of the dot matrix marker to the imaging system can cause a certain shift in the field of view that the dot matrix marker needs to be tested, thereby causing the field of view tested by the dot matrix marker to shift, thereby correcting the position of the test field of view relative to the optical system and preventing the lens or photosensitive chip from shifting.

[0122] After changing the position of the dot matrix marker relative to the optical lens, the image information received by the optical lens is received by the image sensor. By changing the position of the dot matrix marker relative to the optical lens, the information received by the image sensor from the dot matrix marker can be modulated. For example, by moving the dot matrix marker, the pattern of the dot matrix marker can be moved from a 0.8 field of view to a 0.9 field of view, thereby obtaining image information under different fields of view. After changing the information at a predetermined position in the image information, it is then received by the image sensor. Therefore, it is possible to obtain the difference model from the optical lens to the camera module under image stabilization conditions.

[0123] To improve the versatility of image sensors, this application adapts to different resolutions and image plane requirements based on the basic size and shape of the image sensor, its pixel count, and image plane size. In actual production, various situations may arise, such as different projects, different image resolutions for the same project, and different pixel sizes for the same resolution. To ensure the stability and compatibility of the testing environment, this application employs an adaptive image plane adjustment method. This application selects an imaging chip with high pixel count, large image plane, and small pixel units to correspond to different projects and lenses with different requirements, acquires a bitmap image, and then uses an algorithm to adjust the image size and pixel unit size of the acquired bitmap image to obtain point source response information under the testing requirements.

[0124]

[0125] This formula uses the image's subscript to represent the size of the long and wide pixel units in the image, thus enabling the image sensor to achieve images of other sizes and pixel densities by pixel merging or image cropping.

[0126] Another inventive idea in this application is that, for deep learning, in order to achieve high accuracy and compatibility of the network model, in addition to optimizing the network structure and parameters, the training dataset of this application should cover as many possible samples as possible in the project. This ensures the stability and accuracy of the algorithm after the model is ported to this application. However, a specific project may contain millions of samples, and it is impossible for this application to sample each sample to create a dataset. Therefore, this patent proposes a method to improve the quality of the dataset by using the point expansion function during design combined with the point expansion function of actual shooting.

[0127] For a specific project, there will be a theoretical optical design model. This theoretical design model will introduce various aberrations during the assembly of the camera module. Therefore, the products actually delivered to customers are camera modules that contain a certain amount of aberrations. (See attached reference) Figure 6 As shown, according to the manufacturer's big data statistical analysis, the aberrations of the same product basically conform to a normal distribution that fluctuates around the design aberration. Therefore, for a project, as long as the actual point extension function models of the gradient product and the limit product are extracted, a network training dataset covering the product aberrations of that project can be created. Following the above development approach, this method proposes an imaging optimization method, which includes the following steps:

[0128] B1: Provides a deep learning model for image training, and establishes the target set and training set in the deep learning model;

[0129] B2: Provides a set of standard images, which are modulated into a format that can be directly captured by the camera module;

[0130] B3: Test the point spread function of each camera module in a batch of camera modules, obtain the point spread function of each camera module, and establish a normal distribution point spread function model for the batch of camera modules;

[0131] B4: Convolve the standard image set modulated into the format directly captured by the camera module with the normal distribution point spread function model to obtain the training set of the deep learning model;

[0132] B5: Use the standard image set of the format directly captured by the camera module as the target set of the deep learning model, and use the training set to train the deep learning model on the target.

[0133] In application B2, the step of providing a set of standard images and modulating the standard image set into a format directly captured by a camera module includes performing at least one of the following processing methods on the standard image set: shading removal, RI removal, and RGB2RAW processing (RAW format modulation). In this application, to provide image data for deep learning more quickly, pre-stored standard data can be used as the target set in the deep learning model; that is, it is expected that deep learning can be trained on the standard image set as the target.

[0134] A standard image set can be generated using high-definition equipment, such as high-definition SLR cameras, high-definition recording devices, computer-generated bitmaps, and computer-generated engineering drawings. Therefore, a standard image set might be RGB format images processed by an ISP (Image Signal Processor). Thus, during the creation of the image set, the standard image set needs to be de-ISP processed to restore it to a similar level to images directly captured by the camera module. This mainly involves three aspects of processing: shading removal, RI (Relative Intensity) removal, and RGB2RAW (RAW format modulation). If we don't perform de-ISP processing on the standard image set, then the target training set for deep learning will obviously be ISP-processed images—that is, images optimized by image algorithms on top of the captured images—without modulating the standard image set to the format directly captured by the camera module as the target set. This will lead to differences in algorithmic factors during training. As a camera module manufacturer, the ideal scenario is to optimize the images directly captured by the camera module to a high level before providing them to various mobile phone manufacturers for customized algorithmic image synthesis. Even without image algorithm optimization, the quality of the images captured directly by the camera module can still be reflected, so that it can be provided to various mobile phone manufacturers for customized tuning.

[0135] In this application, the shading process includes adjusting the light intensity of the RGB three channels to the design ratio according to the optical system design parameters, as shown in the following formula:

[0136] RxImage = gain r ×Rimage

[0137] GxImage = gain g ×Gimage

[0138] BxImage = gain b ×Bimage

[0139] Among them gain r gain g gain bThese represent the calibration values ​​for r, g, and b brightness, respectively. RxImage, GxImage, and BxImage represent the images after brightness adjustment according to the R, G, and B color channels, respectively. RImage, GImage, and BImage represent the original brightness images under the standard R, G, and B color channels. This method can adjust RGB images into monochromatic light intensity adjustment images according to the three-channel colors, thereby enabling the light intensity of different colors to be displayed separately, ensuring that the brightness of a single-channel color is close to the actual shooting of the camera module.

[0140] RI removal processing involves adjusting the brightness ratio of the RGB three channels according to the different brightness attenuation ratios of the different fields of view in the optical system design parameters. This application also includes adjusting the brightness ratio of the three RGB channels according to the different brightness attenuation ratios of the different fields of view in the optical system design parameters:

[0141] RxImage(i,j)=gainr(i,j)×Rimage(i,j)

[0142] GxImage(i,j)=gaing(i,j)×Gimage(i,j)

[0143] BxImage(i,j)=gainb(i,j)×Bimage(i,j)

[0144] RxImage(i,j), GxImage(i,j), and BxImage(i,j) represent the images after brightness adjustment according to the R, G, and B color channels.

[0145] gainr(i,j), gain(i,j), and gainb(i,j) represent parameters for adjusting the brightness of the R, G, and B color channel images according to different viewing fields.

[0146] Rimage(i,j), Gimage(i,j), and Bimage(i,j) represent the original brightness maps with field of view positions under the standard R, G, and B color channels. This method allows RGB images to be adjusted according to the three color channels into a single-color intensity adjustment format, enabling the display of different color intensities separately. This ensures that the brightness of each single-channel color is close to that of the actual image captured by the camera module. Based on this calculation formula, it is possible to achieve a brightness close to that of the image captured directly by the camera module when adjusting the grayscale images of the three color channels according to the design parameters of the optical system. This ensures that the target set and training set of the deep learning model are at the same level.

[0147] In this application, RGB2RAW processing includes adjusting a standard image set to RAW format, as shown in the following formula:

[0148] According to the formula: RAWImage = RGB2RAW(RGBImage),

[0149] The Raw format is an image directly captured by the camera module. The information stored in the Raw format can have more corresponding space than the conventional JPG format. In particular, the Raw image can have more image channels than the JPG format, thus containing richer information.

[0150] RGB2RAW converts RGB images into RAW images based on the chip's RAW Bayer mode. Since RAW images are raw digital images, they have a wider dynamic range and retain most of the captured image information. RAW images contain richer raw data information, including data on brightness, color, color temperature, and hue. Therefore, the purpose of RAW images as a raw image format is to minimize the loss of information, preserving the data obtained from the sensor and the surrounding captured image (metadata).

[0151] In this application, the step of convolving the standard image set modulated into a camera module directly captured by the camera module with the normal distribution point spread function model to obtain the training set of the deep learning model may further include the following steps;

[0152] Reference Appendix Figure 5 As shown, the extracted optical system aberration PSF is added to a standard image dataset through convolution operations to obtain an input dataset containing aberrations, which can be obtained according to the following formula:

[0153]

[0154] By combining the point spread function difference during lens design with the point spread from actual shooting, model accuracy and compatibility can be improved. Convolving a standard image set with the required point spread function allows for the simulation of images captured by a lens or camera module using that standard image set as a benchmark. This eliminates the need for actual shooting by the lens or camera module, allowing for the generation of simulated images that resemble real-world footage, thus reducing the costs associated with setting up shooting environments. Furthermore, by setting multiple standards and convolving them with the lens's PSF function, multiple simulated real-world images can be obtained, further expanding the variety of datasets required for deep learning model training.

[0155] For deep learning to achieve high accuracy and compatibility in network models, in addition to optimizing the network structure and parameters, the training dataset should cover as many possible samples as possible in the project. This ensures the stability and accuracy of the model after training. However, for a specific project that may contain millions of samples, it is impossible to actually take millions of images to train the model in actual production. On the one hand, the time cost required to take millions of images is high. On the other hand, there will be gradient changes in the production of actual modules. There are good products and there are products that are just within the standard limits. Therefore, it is necessary to select different modules. All of these factors will increase the material cost and time cost.

[0156] This application analyzes the aberration changes caused by component deformation due to manufacturing processes, from lens design to lens assembly into the camera module. After the module undergoes white balance, shading, and RI programming tests, and is processed by the ISP algorithm, image aberrations include algorithmic factors. When shooting with the camera module on a mobile phone, the phone's image algorithm is also introduced. Therefore, it is evident that from the initial lens design of the camera module to its installation on the mobile terminal for shooting, there are many factors that cause image aberrations. After the physical assembly of the camera module, the aberration factors after physical assembly are minimized. By directly encapsulating the algorithm after module assembly, the subsequent mobile phone algorithm is based on the algorithm programming after module assembly. Since the intrinsic aberrations from the physical and optical aspects of the camera module have been compensated, the subsequent mobile phone algorithm can be corrected based on the camera module calibration. This ensures that the mobile phone's algorithm processing module reduces the intrinsic aberrations caused by physical and optical aspects, further improving the image quality presented to the user by the mobile phone.

[0157] For module manufacturers, cost saving is a crucial objective. This application uses big data statistical analysis to show that the aberration distribution of a product basically conforms to the normal fluctuation of the design aberration. Therefore, the products actually delivered to customers all contain a certain aberration distribution. Simulating the training set and target set in a deep learning model can also achieve a similar effect, but without the need for large-scale testing costs.

[0158] Therefore, this patent proposes a method to improve the quality of the training dataset by combining the design point spread function (PSF) and the actual shooting point spread function (PSF). For a specific project, there is a theoretical design point spread function (PSF) in optical design. However, various aberrations are introduced during the design, assembly, programming, and algorithm processes of the camera module. Therefore, the products actually delivered to customers are camera modules containing certain aberrations. According to big data statistical analysis, the aberrations of the same product basically conform to a normal distribution that fluctuates around the design aberration. Therefore, for a project, this application only needs to extract the point spread function (PSF) of a portion of the gradient and limit products, and then combine it with the design point spread function (PSF) to create a network training dataset that covers the aberrations of the project's products. By employing a Point Spread Function (PSF) model for gradient and extreme products for a project, and combining it with a designed PSF, a network training dataset covering the product aberrations of the project can be created. This allows for gradient degradation images to be captured without needing to find standard extreme products for photography. This application can perform PSF model measurements according to standards within specification limits, such as designed standard values, or by classifying products into qualified, excellent, extreme, and defective products. This yields PSF models corresponding to qualified, excellent, extreme, and defective products. Based on the standard classification and the statistically significant normal distribution, the proportions of qualified, excellent, extreme, and defective products are allocated to obtain the corresponding distributions.

[0159] In this application, step S4 - convolving a standard image set modulated into a camera module directly captured by a normal distribution point spread function model to obtain a training set for a deep learning model may further include the following steps;

[0160] The batch of camera modules is divided into four grades: qualified, excellent, extreme, and defective, based on the testing standards. The normal distribution parameters of this batch of camera modules are then obtained according to the testing standards. In this embodiment, the distribution of qualified, excellent, extreme, and defective products already meets the standards for most product scenarios, thus enabling the fitting of a normal distribution curve for the product distribution and obtaining the parameters of the normal distribution.

[0161] Point spread function (PSF) model tests were performed on qualified products, excellent products, extreme products, and defective products respectively, and PSF models of qualified products, excellent products, extreme products, and defective products were obtained respectively. In this embodiment, the distribution of qualified products, excellent products, extreme products, and defective products can meet the standard scenarios of most products. The PSF models of qualified products, excellent products, extreme products, and defective products are obtained for differentiated matching.

[0162] The point spread function (PSF) of qualified, excellent, extreme, and defective products is added to the standard image set and then fused by convolution to form simulated real-world images of the corresponding qualified, excellent, extreme, and defective products. This makes the differences in the training set used for deep learning more obvious, and the gradient distribution of the training set more obvious.

[0163] In another embodiment of this application, as mentioned above, the point spread function (PSF) measured in this solution can obtain different point spread function (PSF) models based on the color temperature, wavelength, and brightness changes of the light source. The point spread function (PSF) models with different mappings can be adjusted to suit the application scenario of the camera module, thereby realizing the training compensation of the deep learning model according to actual needs. For example, if this model of module needs to improve its performance in dim light, then the point spread function (PSF) model under cold light source is used more often for the output of the training set.

[0164] In summary, to improve the adaptability of deep learning models, it is necessary to establish Point Spread Function (PSF) mapping models under different testing environments. Based on the actual shooting requirements of the camera module, the training set should be obtained by appropriately utilizing these PSF mapping models from different testing environments. This can significantly improve the accuracy of deep learning models.

[0165] In summary, one embodiment of this application ensures that the simulated image closely approximates the actual image captured by the camera module. This ensures that the input data for deep learning is images captured with color temperature, wavelength, brightness, etc., of the light source in the actual environment. This guarantees that the input image is always close to the actual product image, thereby improving the accuracy of the model. On the other hand, the output image is based on a standard image set, thus enabling the establishment of a complete method from the input image to the output image.

[0166] On the other hand, after obtaining the product point spread function (PSF) model, the PSF model itself has a certain parameter adjustment space. For example, by introducing factors proportionally, the influence of the PSF model can be increased or decreased, thus obtaining the PSF model of the product in the gradient. The influence of the PSF model can be modified by changing factors according to the proportion of the product in the gradient, thereby affecting the output of the training set. Clearly, this factorization can be achieved using the normal distribution parameters of the camera module.

[0167] This application proposes a method for superimposing a Point Spread Function (PSF) model onto a standard image set to output a simulated image, which mainly includes:

[0168] a) Illuminate the module, acquire the dot matrix image, and extract the brightness components of each channel. Use a camera module or a single-lens optical system to photograph the dot matrix marker, acquiring the corresponding dot matrix image. Similar to the previous test system, this will not be elaborated upon here.

[0169] b) Extract Point Spread Function (PSF) data for each field of view. For the acquired image, extract the PSF model for each field of view at a predetermined location, such as the important 0.8 field of view.

[0170] c) Point Spread Function (PSF) Data Preprocessing. Preprocessing of the PSF data includes classifying the PSF data according to a specific sample and gradient. This allows for the differentiation of PSF models within the same batch of products based on a normal distribution, thus obtaining the gradient distribution of the camera modules in that batch. PSF preprocessing also includes PSF sampling frames, which represent image segmentation. This segmentation involves dividing the entire image into squares with a uniform number of pixels. Within these squares, the distance between test blocks is determined, along with the step size and size of the test blocks. This determines how many test blocks to select to fill the image. This method limits the resolution of the image, ensuring that the number and size of image tests meet the requirements.

[0171] In addition, in different embodiments, testing is performed according to a fixed test image. On the one hand, this reduces the number of pixel blocks required for testing, and on the other hand, it increases the amount of testing required, thereby enabling more measurement methods to be obtained for measuring the images between the test blocks. In short, if a sampling frame is used, and the selected sampling frame is a matrix with a length and width of 100*100, by transforming the matrix into a 50*50 pixel matrix, it is possible to take every other pixel. In this case, the step size of 50*50 is 2. If this application only takes the middle 50*50 of the 100*100, then the step size is 1. By refining the sampling frame, the test blurring caused by the original sampling frame being too large can be corrected by making the test points within the frame more accurate after the sampling frame is reduced, thereby ensuring better measurement accuracy.

[0172] d) On the other hand, standard images are acquired. Standard images are high-definition images and can be considered as the target output images of deep learning models. Standard images do not have aberrations or color casts.

[0173] d1) Preprocess the standard image. For example, preprocess the standard image to fit the imaging surface size of the camera module being tested, according to the size requirements. In addition, preprocessing the standard image includes restoring the standard image to the direct output image of the camera module before image processing.

[0174] In this application, the preprocessing of the standard image includes deshading, RI removal, and white balance adjustment, thereby obtaining a restored image that approximates the quality of the standard image obtained directly from a camera module.

[0175] Shadow removal primarily involves restoring lens shading, which is similar to the previously mentioned aspects. Here, we'll supplement this with information on shadow removal, which includes two main types: Luma Shading (brightness uniformity) and Color Shading (color uniformity). Luma Shading is a commonly used technique in the industry to correct vignetting. When an image exhibits a brighter central area and a darker periphery, brightness uniformity is compensated for during the camera module manufacturing process. Additionally, color shading (color uniformity) manifests as color inconsistency between the center and surrounding areas of an image, meaning a color cast occurs in either the center or the periphery. Color uniformity is compensated for during the camera module manufacturing process. Shadow removal restores the standard image to its original state as directly captured by the camera module. This ensures that during model training, both the input and output images are directly captured by the camera module, rather than images compensated by algorithms. Therefore, when the input and output images are in the same state, the deep learning model can be trained on the same image. This allows the model to directly determine the optimization type and optimize the internal network of the deep learning model when the images are in the same state.

[0176] d2) Convert the standard image to a RAW image. A RAW image is a raw digital image with a wider dynamic range. It retains most of the captured image information and contains richer raw data, including data on brightness, color, color temperature, and hue. Therefore, the purpose of using RAW as the raw image format is to minimize information loss, preserving the data from the sensor and the surrounding captured image (metadata).

[0177] Reference Appendix Figure 7 As shown in the flowchart, this application provides a method for manufacturing a camera module, characterized by comprising the following steps:

[0178] C1: Provides an image sensor, a lens, a dot matrix marker, a deep learning model, and a processor module;

[0179] On the one hand, the deep learning model of this application needs to be loaded into the processor for use. On the other hand, after assembling the photosensitive chip and lens into a camera module, one purpose of this application is to optimize the image data obtained by the camera module with the help of the deep learning model. The deep learning model can improve various defects in the camera module (such as poor motor orientation tilt leading to poor image plane tilt, field curvature caused by chip warping due to baking, etc.). It provides camera module components including photosensitive chip, lens, motor, or lens mount for subsequent assembly / setting in preset positions for shooting, or provides a method of already assembling into a camera module to achieve shooting. Under the premise of not affecting the shooting of the test camera module, providing steps including photosensitive chip and lens can also be considered as providing all the components of the test camera module of this model.

[0180] C2: The photosensitive chip and the lens are set in a preset position to form an image-capable camera module system;

[0181] To ensure that the test dataset closely resembles actual manufacturing processes and includes factors that cause image orientation degradation in actual manufacturing, the process of setting the image sensor and lens at a preset position to form an image-capable camera module system can also be considered as assembling the camera module according to a prescribed manufacturing process. In this application, "setting at a preset position" can also refer to assembling or simply placing the lens at a preset position above the image sensor to simulate the process of taking a picture.

[0182] C3: The photosensitive chip captures the dot matrix label through the optical lens, and obtains the dot matrix label information captured by the imageable camera module system;

[0183] The step of acquiring the dot matrix marker information captured by the image-capable camera module system through the optical lens and the image sensor chip includes: extracting the point source response information of each point on the dot matrix marker, extracting the intensity information map of each color channel of a single point source, and extracting the dot spread function information of each color channel of a single point source. The corresponding functions and effects are described in the preceding section on the classification of image channels.

[0184] In order to make the test dataset closer to actual production and manufacturing, and to include factors that cause image orientation degradation in actual production and manufacturing, after setting the photosensitive chip and the lens in a preset position to form an image-capable camera module, the image actually captured by the photosensitive chip can be considered to be the normal performance of this model of camera module.

[0185] C4: Based on the dot matrix target information captured by the imageable camera module system, establish the response of the imageable camera module to the point source;

[0186] The point source response (point spread function) mentioned above represents the optical degradation factors of the image captured by the image-capable camera module system when the target plate is a dot matrix.

[0187] C5: The deep learning model uses the response of the imageable camera module to a point source to establish a compensation model for the imageable camera module, and updates the deep learning model according to the compensation model;

[0188] The step of updating the deep learning model by using the imageable camera module's response to a point source to establish a compensation model for the imageable camera module includes: providing a standard image and convolving the standard image with the point spread function of each color channel of the single point source to form a simulated real-world image, which is then used as the input to the deep learning model. The corresponding functions and effects are described above in the section on simulated real-world images.

[0189] In the step of updating the deep learning model by using the response of the imageable camera module to a point source to establish a compensation model for the imageable camera module, a standard image is also used as the output of the model, and the deep learning model is trained with the goal of optimizing the input to the output. The corresponding functions and effects are described in the preceding section on deep model training.

[0190] The deep learning model utilizes the response of the imageable camera module to a point source to establish a compensation model for the imageable camera module. The step of updating the deep learning model further includes: extracting point source response information of each point in the standard image; extracting intensity information maps of each color channel of the single point source in the output end; extracting the dot spread function information of each color channel in the output end; and establishing a point response difference model from the optical lens to the optical system assembled into the camera module by combining the dot spread function information of each color channel in the input end and the dot spread function information of each color channel in the output end.

[0191] The step of updating the deep learning model by using the response of the imageable camera module to a point source to establish a compensation model for the imageable camera module, and further including: establishing a compensation model based on the point response difference model, and updating the deep learning model according to the compensation model. Refer to the description of the compensation model described above.

[0192] The compensation model includes a mapping between gradient performance metrics and point expansion function sets within a batch of camera modules, which enables the deep-trained model to optimize the imaging of camera modules within a batch.

[0193] The deep learning model described above takes the point source response captured by the camera module and the convolution of the standard image as input, and the standard image as output. The deep learning model is trained with the standard image as the target, in order to obtain a compensation model that compensates the input as the output. In fact, the compensation model is the deep learning model after updating the parameters or network structure.

[0194] C6: Load the updated deep learning model into the processor module;

[0195] By loading the updated deep learning model onto the processor module, the requirement of this application for the deep learning model to be loaded into the processor for use can be fulfilled.

[0196] C7: Integrate the processor module into the image-capable camera module;

[0197] The process of providing the image-capable camera module with the processor module enables the deep learning model of this application to be invoked by the camera module. The processor module can be a chip integrated into the camera module or the computing processor (CPU) of the mobile phone. In this application, it is preferred that the camera module has an integrated processor module, which enables the image quality directly output by the camera module to the mobile terminal to be relatively high, thereby facilitating the mobile phone to output higher quality images.

[0198] Reference Appendix Figure 8 As shown in the flowchart, this application provides an imaging optimization method for a camera module, characterized by comprising the following steps:

[0199] D1: Obtain the mapping data of the batch camera modules with respect to point responses after gradient classification;

[0200] As described in the previous section, a batch of camera module products contains graded and extreme products due to different performance. The manufacturer's main goal is to improve production yield. Optimizing graded and extreme products within the performance requirements can improve production efficiency and prevent defective products from flowing into the next process, thus avoiding waste.

[0201] The step of obtaining the mapping data of the batch camera modules regarding the point response after gradient classification can be further divided into the following steps:

[0202] D1.1: Classify the batch of camera modules according to their performance indicators;

[0203] The batch of camera module products described above includes defective products (products outside the performance specifications), extreme products (products close to or slightly exceeding the performance specifications), and gradient products (products within the performance specifications but with a high-to-low performance distribution). The manufacturer's primary objective is to improve production yield. Defective products often include those with extremely poor performance, which are generally not correctable by later-stage deep learning models. This solution first performs gradient classification on gradient and extreme products, including fitting the performance indicators and distribution ratios according to the proportion of products exhibiting a normal distribution. For example, in this batch of camera modules, the proportion of products with an SFR of 50 is 1%, and the proportion of products with an SFR of 60 is 2%, ultimately obtaining a gradient based on the performance indicators and distribution ratios.

[0204] D1.2: Obtain the mapping data of test parameters and point responses of batch camera modules after gradient classification;

[0205] Within this batch of products, multiple products are selected according to performance indicators and distribution ratios to characterize the overall performance of the batch. For example, if the batch contains 100k camera modules, after gradient classification, products with a surface response rate (SFR) of 50 account for 1% and those with an SFR of 60 account for 2%. This allows for the selection of 10 camera modules with an SFR of 50 and 20 with an SFR of 60. By selecting camera modules in appropriate proportions based on different performance indicators, a small batch of samples representing the overall batch performance can be obtained. These small batches of samples are then tested to obtain point response data, which is mapped to the gradient data of the camera modules to obtain point response data for products at different performance levels.

[0206] D2: Obtain the point extension function set for gradient classification based on the mapping data of the batch camera modules with respect to point responses after gradient classification;

[0207] In the preceding section, point response data for products under different performance levels were obtained, establishing a mapping between performance and point response. Specifically, this allows us to determine the point response performance of a product with an SFR of 50, as well as the point response performance of a product with an SFR of 60. After measuring a small batch of samples from the gradient classification module, we can obtain the point extension function set for gradient classification. This set of point extension functions can characterize the entire batch of products within the performance metrics, and also represents the similar performance of all products in that batch.

[0208] D3: Obtain a standard image set and use it as the output of a deep learning model;

[0209] As described above, the standard image, as an aberration-free image, is used as the target for training the deep learning model. The specific order of this step may not precede S4; please refer to the appendix for details. Figure 8 The sequence of actions.

[0210] D4: Combine the gradient classification point expansion function set with the standard image set through convolution to form the training image set;

[0211] By convolving the point spread function set representing all products in the batch with a standard image set, a simulated real-world image of the batch of products can be obtained, which can then be used as input data for training a deep learning model. This approach can significantly improve the cost of testing, reduce the time required for testing, and lower overall costs.

[0212] The step of convolving the point expansion function set of gradient classification with the standard image set to form the training image set may also include S4.1: the standard image set selects the point expansion function set of test parameter type according to the training requirements and performs convolution.

[0213] The standard image set, selected based on training requirements, uses a point spread function set with different test parameter types for convolution. This allows for model training as needed. In reality, model training involves setting numerous parameters, and modifying certain parameters can more specifically address product defects. For example, to correct color cast issues in image sensors, it's necessary to improve the performance of camera modules under cool color temperatures. Different test types can be selected for different tests, enabling simulated shooting of images from various input sources to achieve more complex, general-purpose models.

[0214] D5: Use the training image set as the input to the deep learning model;

[0215] The deep learning model described above uses the point source response captured by the camera module and the convolution of the standard image as input. The deep learning model uses the standard image as output. This allows the deep learning model to use a training image set that simulates real shooting as input and an image with near-aberration-free image as output. The deep learning model can continuously find defects and optimize from input to output. Finally, through continuous training, the model can be updated and a compensation model can be obtained.

[0216] D6: Deep learning models are trained by optimizing the input into the output based on the target.

[0217] The deep learning model described above takes the point source response captured by the camera module and the convolution of the standard image as input, and the standard image as output. The deep learning model is trained with the standard image as the target, in order to obtain a compensation model that compensates the input as the output. In fact, the compensation model is the deep learning model after updating the parameters or network structure.

[0218] D7: The trained deep learning model is loaded into the processor module;

[0219] The deep learning model described above uses the point source response captured by the camera module and the convolution of a standard image as input, and the standard image as output. The deep learning model is trained using the standard image as the target, with the aim of ultimately obtaining a compensated model that compensates the input to become the output. In reality, this compensated model is a deep learning model with updated parameters or network structure. The step of providing the imageable camera module with access via the processor module completes the process of providing the camera module with access to the deep learning model described in this application.

[0220] D8: Optimize the deep learning model by calling the trained model based on the actual images captured by the camera module;

[0221] Using the final optimized deep learning model, images directly captured by the camera module can be optimized to output images with better resolution.

[0222] The processor module can be a chip integrated into the camera module or the mobile phone's computing processor (CPU). In this application, it is preferred that the camera module has an integrated processor module, which enables the camera module to output higher quality images directly to the mobile phone terminal, thereby facilitating the mobile phone to output higher quality images.

[0223] The basic principles, main features, and advantages of this application have been described above. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely the principles of this application. Various changes and modifications can be made to this application without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claims. The scope of protection claimed by this application is defined by the appended claims and their equivalents.

Claims

1. An imaging optimization method for a camera module, characterized in that, Includes the following steps: The object is photographed using an optical lens to obtain an image of the object photographed by the optical lens; An optical system response model of the optical lens is established based on the image of the object captured by the optical lens; Based on the optical system response model of the optical lens, a difference term including optical system aberrations is added to obtain an aberration-containing model input terminal, which includes differentiating the response of the camera module's optical system to a point source and the response of the optical lens's optical system to a point source into a difference term of optical system aberrations. The deep learning model is trained based on the aberration-containing model input to obtain the optical system response difference model; The optical system response difference model is used to optimize images captured by the camera module. The step of training the deep learning model to obtain the optical system response difference model based on the aberration-containing model input further includes the following steps: Using the response of the optical system of the optical lens to a point source as the input and the response of the optical system of the camera module to a point source as the output, a point response difference model is established from the optical lens to the optical system assembled into the camera module based on the difference in the responses of the same point source from the input to the output.

2. The imaging optimization method for the camera module according to claim 1, wherein, The step of adding a difference term including optical system aberrations to the optical system response model of the optical lens to obtain the aberration-containing model input also includes the following steps: The difference in optical system aberrations includes factors that cause wavefront aberrations in the optical system, arising from the design, production, manufacturing, and use of the camera module.

3. The imaging optimization method for the camera module according to claim 2, wherein, The step of acquiring an image of an object by photographing it through an optical lens also includes the following steps: The dot matrix marker is photographed using an optical lens to obtain an image of the dot matrix marker captured by the optical lens.

4. The imaging optimization method for the camera module according to claim 3, wherein, The step of establishing an optical system response model of the optical lens based on the image of the object captured by the optical lens also includes the following steps: The optical system of the optical lens is established to respond to a point source based on the image of a dot matrix target taken through the optical lens.

5. The imaging optimization method for the camera module according to claim 4, wherein, The step of adding a difference term including optical system aberrations to the optical system response model of the optical lens to obtain the aberration-containing model input also includes the following steps: The optical lenses are assembled into a camera module according to the specified manufacturing process; The dot matrix marker is photographed by the camera module to obtain an image of the dot matrix marker captured by the camera module; The response of the camera module's optical system to the point source is established based on the image of the dot matrix target taken by the camera module; The differences between the optical system response of the camera module to a point source and the optical system response of the optical lens to a point source are processed into a difference item from the optical lens to the camera module. The optical system response model of the optical lens is used as a basis to add a difference term from the optical lens to the camera module.

6. The imaging optimization method for a camera module according to claim 5, wherein, The step of optimizing the image captured by the camera module using the optical system response difference model further includes the following steps: The point response difference model of the optical system from the optical lens to the optical system assembled into the camera module is used to optimize the images captured by the camera module.

7. The imaging optimization method for a camera module according to claim 6, wherein, The difference term including optical system aberrations includes aberrations caused by the optical system design of the camera module in near-focus and far-focus images.

8. The imaging optimization method for the camera module according to claim 7, wherein, The difference item including optical system aberrations includes aberrations in the optical system of the camera module caused by process factors during the camera module assembly process.

9. The imaging optimization method for the camera module according to claim 8, wherein, The difference item including optical system aberrations includes aberrations in the camera module's optical system caused by component batch factors.

Images