Imaging control system, method, image sensor, processor, storage medium, and program product

Abstract

The invention relates to an imaging control system and method, an image sensor, a processor, a computer readable storage medium and a computer program product. The imaging control system includes an image sensor and a processor. The image sensor comprises a pixel array with a plurality of pixel points and a spatial light modulation array with a plurality of light modulation units. The spatial light modulation array is located on the light sensing side of the pixel array, and the light modulation unit corresponds to at least one pixel. The processor is configured to independently adjust control parameters of the light modulation units in the spatial light modulation array, so that the adjusted light modulation units attenuate polarized light in incident light.

Description

Technical Field

[0001] This application relates to the field of sensor technology, and in particular to an imaging control system, an imaging control method, an image sensor, a processor, a non-volatile computer-readable storage medium, and a computer program product. Background Technology

[0002] In real-world scenarios, light typically includes both natural light and reflected light. Reflected light is light obtained after being reflected from various surfaces. Due to its partially polarized nature, it can cause numerous inconveniences in production and daily life. For example, when taking photos / imaging images in an environment with polarized reflected light, the presence of reflected light in the light entering the lens causes a glare effect in the captured image. This can easily obscure the true scene content, resulting in poor image quality. Summary of the Invention

[0003] Therefore, it is necessary to provide an imaging control system, method, image sensor, processor, computer-readable storage medium, and computer program product that can eliminate or at least partially attenuate reflected light, in order to address the above-mentioned technical problems.

[0004] On one hand, this application provides an imaging control system, including:

[0005] An image sensor, comprising a pixel array and a spatial light modulation array; wherein the pixel array comprises a plurality of pixels; the spatial light modulation array is located on the photosensitive side of the pixel array, and the spatial light modulation array comprises a plurality of light modulation units, each light modulation unit being configured corresponding to at least one pixel; and

[0006] The processor, connected to the image sensor, is configured to independently adjust the control parameters of each of the optical modulation units in the spatial light modulation array, such that the adjusted optical modulation units attenuate the polarized light in the incident light.

[0007] In some embodiments, the optical modulation unit includes:

[0008] A modulation subunit, connected to the processor, and

[0009] A polarizer is located on the side of the light modulation unit closer to the pixel array.

[0010] In some embodiments, the image sensor further includes a microlens array; wherein the microlens array is located on the side of the spatial light modulation array away from the pixel array, or the microlens array is located between the spatial light modulation array and the pixel array.

[0011] In some embodiments, the processor is further configured to: change the control parameters of each of the optical modulation units multiple times and perform pre-capture using the image sensor; perform polarization analysis based on the pre-captured images to determine target control parameters for each of the optical modulation units; use the optical modulation units controlled by the target control parameters to attenuate the polarized light in the incident light; control the corresponding optical modulation units through the target control parameters and perform actual capture using the image sensor.

[0012] In some embodiments, the control parameters of the optical modulation unit are used to control the deflection angle corresponding to the corresponding optical modulation unit, and the processor is further configured to: determine a first deflection angle applicable to the pixel based on pixel information corresponding to the same pixel in multiple pre-captured images; for any optical modulation unit, determine a second deflection angle corresponding to the optical modulation unit based on the first deflection angle applicable to the pixel corresponding to the optical modulation unit; and determine the target control parameters of the corresponding optical modulation unit according to the second deflection angle corresponding to each optical modulation unit.

[0013] In some embodiments, the processor is further configured to: for any pixel, perform function fitting based on pixel information corresponding to the pixel in multiple pre-captured images, and determine a first deflection angle that minimizes the pixel information corresponding to the pixel based on the fitted function.

[0014] In some embodiments, the processor is further configured to: for any optical modulation unit, determine at least one pixel corresponding to the optical modulation unit based on the correspondence between the optical modulation unit and the pixel; and fuse the first deflection angle corresponding to the determined at least one pixel to obtain a second deflection angle corresponding to the optical modulation unit.

[0015] In some embodiments, the processor is further configured to: smooth the second deflection angles corresponding to all optical modulation units in the spatial light modulation array to obtain a third deflection angle corresponding to each optical modulation unit; and determine the target control parameters of the corresponding optical modulation unit based on the third deflection angle corresponding to each optical modulation unit.

[0016] In some embodiments, the processor is further configured to: for each optical modulation unit, change the control parameters of the optical modulation unit while keeping the control parameters of other optical modulation units unchanged, and obtain multiple images of the optical modulation unit under different control parameters; for each optical modulation unit, based on the pixel information corresponding to the same pixel in the obtained multiple images, filter out the pixels affected by the optical modulation unit, and associate the pixels affected by the optical modulation unit with the optical modulation unit; and obtain the correspondence between the optical modulation unit and the pixels according to the pixels associated with each optical modulation unit in the spatial optical modulation array.

[0017] In some embodiments, the processor is further configured to: for each optical modulation unit, determine the pixel statistics of each pixel based on the pixel information corresponding to the same pixel in the multiple images obtained, and take the pixels whose pixel statistics satisfy a preset difference condition as the pixels affected by the optical modulation unit.

[0018] In some embodiments, the processor is further configured to: if any pixel is associated with multiple optical modulation units, select one optical modulation unit from the multiple optical modulation units associated with it as the final associated optical modulation unit; if any pixel does not have an associated optical modulation unit, select one optical modulation unit to associate with the pixel based on the optical modulation units associated with the pixels surrounding the pixel.

[0019] On the other hand, this application provides an imaging control method for an image sensor, the image sensor including a pixel array having multiple pixels and a spatial light modulation array having multiple light modulation units, the spatial light modulation array being located on the photosensitive side of the pixel array, each light modulation unit being configured corresponding to at least one pixel; the method includes: independently adjusting the control parameters of each light modulation unit in the spatial light modulation array, so that the adjusted light modulation unit attenuates the polarized light in the incident light.

[0020] In some embodiments, the method specifically includes: repeatedly changing the control parameters of each of the optical modulation units and taking a pre-shot using the image sensor; performing polarization analysis based on the pre-shot image to determine the target control parameters of each of the optical modulation units; using the optical modulation units controlled by the target control parameters to attenuate the locally polarized light in the incident light; controlling the corresponding optical modulation units through the target control parameters and taking an actual shot using the image sensor.

[0021] On the other hand, this application also provides an image sensor, including:

[0022] A pixel array, the pixel array comprising a plurality of pixels;

[0023] A spatial light modulation array is located on the photosensitive side of the pixel array. The spatial light modulation array includes multiple light modulation units, and each light modulation unit is configured corresponding to at least one pixel.

[0024] The optical modulation unit can be independently controlled to modulate the polarization state of the incident light. The optical modulation unit attenuates the polarized light in the incident light to different degrees depending on the polarization state of the incident light.

[0025] In some embodiments, the optical modulation unit includes:

[0026] Modulation subunit, and

[0027] A polarizer is located on the side of the light modulation unit closer to the pixel array.

[0028] In some embodiments, the modulation subunit includes a liquid crystal-based modulation subunit or an electro-optic crystal-based modulation subunit.

[0029] In some embodiments, the image sensor further includes:

[0030] A microlens array having a plurality of microlenses, each microlens corresponding to at least one of a plurality of pixels; wherein the microlens array is located on the side of the spatial light modulation array away from the pixel array, or the microlens array is located between the spatial light modulation array and the pixel array.

[0031] On the other hand, this application also provides a processor including logic circuitry capable of executing a computer program stored in a memory, wherein the logic circuitry, when executing the computer program, implements the steps of the imaging control method for an image sensor described above.

[0032] On the other hand, this application also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the imaging control method for an image sensor described above.

[0033] On the other hand, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the imaging control method for an image sensor described above.

[0034] The aforementioned imaging control system, imaging control method, image sensor, processor, computer-readable storage medium, and computer program product arrange a spatial light modulation array (especially a spatial light polarization modulation array) on the photosensitive side of the pixel array of the image sensor to modulate the polarization state of the incident light. This spatial light modulation array includes multiple light modulation units, each corresponding to at least one pixel and independently controllable. The processor independently controls the control parameters of the multiple light modulation units in the spatial light modulation array, enabling adaptive attenuation of locally polarized light in the incident light. This allows the light signal entering the image sensor to eliminate energy in the main polarization direction at different locations as much as possible, thereby weakening or even eliminating reflected light while retaining the object information originally obscured by the reflected light. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is an exemplary schematic diagram of the reflection and refraction of light in the environment;

[0037] Figure 2 This is a schematic diagram of the structure of an image sensor according to an embodiment of this application;

[0038] Figure 3 This is a schematic diagram of an image sensor and an example optical path according to an embodiment of this application;

[0039] Figure 4A This is a top view of an image sensor with a spatial light modulation array according to an embodiment of this application;

[0040] Figure 4B This is a top view of an image sensor with a spatial light modulation array according to another embodiment of this application;

[0041] Figure 5A This is an exemplary cross-sectional view of an image sensor according to an embodiment of this application;

[0042] Figure 5B This is an exemplary cross-sectional view of an image sensor according to another embodiment of this application;

[0043] Figure 6A This is an exemplary cross-sectional view of a modulation subunit according to an embodiment of this application;

[0044] Figure 6B This is an exemplary cross-sectional view of a liquid crystal-based modulation subunit according to an embodiment of this application;

[0045] Figure 7 This is an architectural diagram of an imaging control system according to an embodiment of this application;

[0046] Figure 8 This is a flowchart of an imaging control method according to an embodiment of this application;

[0047] Figure 9 This is a schematic flowchart of the pre-shooting process in the imaging process according to an embodiment of this application;

[0048] Figure 10 This is a flowchart illustrating the steps of determining the target control parameters of each optical modulation unit based on a pre-captured image according to an embodiment of this application.

[0049] Figure 11 This is a schematic diagram of the coordinate system of the optical modulation unit according to an embodiment of this application;

[0050] Figure 12A This is a flowchart of the spatial light modulation array calibration steps according to an embodiment of this application;

[0051] Figure 12B This is a flowchart of the spatial light modulation array calibration steps according to another embodiment of this application;

[0052] Figure 13 This is a structural block diagram of an imaging control device according to an embodiment of this application.

[0053] Explanation of reference numerals in the attached figures:

[0054] 10-Imaging control system, 20-Image sensor, 30-Processor, 40-Object to be photographed, 50-Lens, 21-Pixel array, 22-Spatial light modulation array, 23-Microlens array, 24-Surface covering device, 210-Pixel, 220-Light modulation unit, 221-Modulation subunit, 222-Polarizer, 223-Glass, 2211-Electrode layer, 2212-First alignment layer, 2213-Nematic liquid crystal, 2214-Second alignment layer. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0057] It should be noted that the terms "first," "second," etc., mentioned in this application may be used to describe various objects (such as components and / or data), but these objects are not limited by these terms. These terms are only used to distinguish one object from another and are not used to limit the order.

[0058] Furthermore, when used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having,” etc., specify the presence of the stated features, integrals, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, integrals, steps, operations, components, parts, or combinations thereof. Meanwhile, in this specification, the term “and / or” includes any and all combinations of the associated listed items.

[0059] Before describing the embodiments of this application in detail, the relevant technical knowledge involved in this application will be described first:

[0060] Light waves are transverse waves, and the vibration directions of their electric and magnetic fields are perpendicular to their propagation direction, thus exhibiting polarization. The polarization direction of light refers to the vibration direction of its electric vector. The vibration direction of the electric vector of natural light is usually random, therefore its macroscopic degree of polarization is zero. After natural light undergoes reflection under certain conditions, its vibration direction changes. When light enters water or other media from air, refraction and reflection occur, causing a change in the polarization direction of the light, resulting in partially polarized light.

[0061] Please refer to Figure 1 , Figure 1 This is an exemplary schematic diagram of the reflection and refraction of light in an environment. Light is emitted from a light source and propagates in air (medium 1, refractive index n1). After reflection and refraction at the surface of a medium (medium 2, such as a surface of water, refractive index n2), the reflected light is received by a camera, as shown below. Figure 1 As shown, taking the light path from air (n1) to the water surface (n2) as an example ( ).

[0062] Take the interface normal as Axis, take The axis lies in the plane of incidence, thus The axis is perpendicular to the incident plane. This forms a right-handed orthogonal system. Let the angle of incidence, angle of reflection, and angle of refraction be respectively... , and According to the laws of reflection and refraction, , Furthermore, the point vector components in each beam are defined: [Define...] , and Let be the unit vectors of the incident, reflected, and refracted light rays respectively, along the direction of light propagation; define , and The incident light, reflected light, and refracted light are respectively located perpendicular to the incident plane and along... Unit vector in the direction; definition , and Let be the unit vectors of the incident, reflected, and refracted light rays respectively, in directions parallel to the incident plane, and let be the corresponding vectors of the incident, reflected, and refracted light rays respectively. , and Conforms to the right-handed system, that is , , In optics, based on the boundary value relations of electromagnetic fields, the components of the incident and reflected fields at points near each other on both sides of the interface satisfy the following:

[0063]

[0064] in, The p-component of the electric field of the reflected light; The p-component of the electric field of the incident light; The s-component of the electric field of the reflected light; This refers to the s-component of the electric field of the incident light. Looking at the s and p-components of the electric field of the reflected light, if the corresponding incident light is natural light (…),… = After reflection, the s and p components of the electric field are no longer in equilibrium. ≠ This demonstrates polarization characteristics.

[0065] Furthermore, it is understandable that when a light source emits (natural light) rays in different directions, these rays can be reflected by different media. In this case, the angle of incidence corresponding to the reflection of rays from different directions is different. Simultaneously, because the PSK coordinate system is a local coordinate system defined on each ray, the resulting distribution of polarized light is also different relative to the camera coordinate system. That is, when the camera captures an image, the light that passes through the lens and forms an image at different positions on the pixel plane may come from different positions and directions, thus resulting in different polarization patterns.

[0066] In related technologies, there are generally two methods to address this type of high-intensity reflected light. One method is to use HDR (High Dynamic Range) technology, employing ultra-short exposure times (and / or ultra-low gain, and / or extremely low sensor sensitivity) to minimize the area of ​​high-brightness reflected light in a short exposure. However, this method usually significantly reduces image contrast, and other details within the high-brightness area are greatly compressed or even erased. The other method utilizes the partial polarization characteristics of reflected light, using a global linear polarizer before imaging, ensuring that the direction of minimum vibration of the reflected light aligns with the transmission direction of the linear polarizer, thereby reducing the energy of the strong vibration direction of the reflected light in the camera image. However, as mentioned earlier, according to polarization optics theory, natural light has different polarization directions after reflection from different planes (different incident angles, different reflection planes, different reflection angles). For example, the polarization directions of light reflected from oncoming headlights on the water surface in rainy weather and the light entering the vehicle's front-view camera after reflection from the side of a roadside curb (a wet plane) are not the same. Therefore, a single global (same direction) linear polarizer cannot automatically adapt to different polarization conditions in the same image.

[0067] In response to the above situation, this application provides an improved solution, which uses a processor to independently control the control parameters of multiple optical modulation units in a spatial light modulation array, enabling them to adaptively attenuate locally polarized light. This allows the light signal entering the image sensor to eliminate energy in the main polarization direction at different locations as much as possible, thereby achieving the purpose of weakening or even eliminating reflected light while retaining the object information originally obscured by reflected light.

[0068] In one exemplary embodiment, an image sensor is provided. Please refer to... Figure 2 , Figure 2 The structure of an image sensor 20 is schematically shown. The image sensor 20 includes a pixel array 21 and a spatial light modulation array 22. The pixel array 21 includes a plurality of pixels 210. The spatial light modulation array 22 is located on the photosensitive side of the pixel array 21 and includes a plurality of light modulation units 220, each light modulation unit 220 corresponding to at least one pixel 210. The light modulation units 220 can be independently controlled to modulate the polarization state of incident light; different polarization states of the incident light result in different degrees of attenuation of the polarized light by the light modulation units 220.

[0069] An image sensor is a semiconductor device that converts optical images into electronic signals (usually digital signals). It is a core component of digital imaging systems such as cameras, smartphone cameras, surveillance cameras, and medical imaging equipment. The pixel array is the core component of an image sensor, consisting of a large number of photosensitive units (such as pixels). Each pixel can contain a photodiode, which converts incident photons (light) into electrons (charge).

[0070] The embodiments of this application construct the main body of the image sensor 20 by adding a spatial light modulation (SLM) array 22 to the photosensitive side of the pixel array 21. The photosensitive side of the pixel array 21 is the side that receives incident light, i.e., the side closer to the lens. For example, please refer to... Figure 3 , Figure 3 This is a schematic diagram of an image sensor 20 and an example optical path according to an embodiment of this application. Figure 3 As shown, light from the subject 40 is focused by the lens 50 and captured by the image sensor 20.

[0071] Optionally, the spatial light modulation array 22 can specifically be a spatial light polarization modulation array or a spatial light polarization modulation array (SLPM array) for modulating the polarization state of incident light. It can be understood that the spatial light modulation array 22 is an array composed of multiple optical modulation units 220, each of which has spatial light polarization / polarization modulation function and can be independently controlled. Optionally, the optical modulation unit 220 can specifically be a spatial light polarization / polarization modulation unit (SLPM unit).

[0072] Furthermore, each optical modulation unit 220 may be configured corresponding to at least one pixel 210 for adjusting (e.g., eliminating or at least reducing) the polarization component in the incident light incident on the at least one pixel 210.

[0073] In some embodiments, an optical modulation unit 220 is configured corresponding to at least one pixel 210. This can be understood as an optical modulation unit 220 covering at least one pixel 210 in the optical path (e.g., covering one pixel 210, or covering M*N pixels 210). In other words, the optical modulation unit 220 can provide a mask based on pixels 210. For example, please refer to... Figure 4A and Figure 4B , Figure 4A and Figure 4B This is a top view of an image sensor 20 with a spatial light modulation array 22. Figure 4A and Figure 4B In this configuration, the spatial light modulation array 22 is generally aligned with the pixels of the image sensor 20. In some cases, such as... Figure 4A As shown, a single light modulation unit 220 (such as a gray square) can cover a single pixel 210 of the image sensor 20 (such as a black square covered by a gray square); in other cases, such as Figure 4B As shown, a single light modulation unit 220 (such as a gray square) can cover multiple (exemplarily, such as 4*4) pixels 210 of the image sensor 20 (such as 4*4 black squares covered by a single gray square).

[0074] The aforementioned optical modulation unit can be independently controlled to modulate the polarization state of the incident light. A change in the polarization state of the incident light means that the degree of attenuation of the polarized light in the incident light by the optical modulation unit also changes.

[0075] The aforementioned image sensor arranges a spatial light modulation array (especially a spatial light polarization modulation array) on the photosensitive side of its pixel array to modulate the polarization state of the incident light. This spatial light modulation array includes multiple light modulation units, each corresponding to at least one pixel and independently controllable. This allows for adaptive attenuation of locally polarized light in the incident light, thereby minimizing the energy of the main polarization direction at different locations in the light signal entering the image sensor. This achieves the goal of weakening or even eliminating reflected light while preserving the object information originally obscured by the reflected light.

[0076] In some embodiments, the image sensor 20 further includes a microlens array 23. The microlens array 23 has a plurality of microlenses, each microlens being disposed corresponding to at least one pixel. The microlens array 23 is located on the side of the spatial light modulation array 22 away from the pixel array 21, or the microlens array 23 is located between the spatial light modulation array 22 and the pixel array 21.

[0077] In some embodiments, reference Figure 5A The microlens array 23 is located on the side of the spatial light modulation array 22 away from the pixel array 21; that is, the spatial light modulation array 22 can be arranged between the pixel array 21 and the microlens array 23. When the spatial light modulation array 22 is arranged between the pixel array 21 and the microlens array 23 of the image sensor 20, as a conventional component of the image sensor 20, the outermost surface of the image sensor 20 can be provided with a surface covering device 24, generally made of glass or high-transparency plastic, to protect the interior of the image sensor 20. Figure 5A As shown.

[0078] In other embodiments, reference is made to... Figure 5BThe microlens array 23 is located between the spatial light modulation array 22 and the pixel array 21; that is, the spatial light modulation array 22 can be arranged on the side of the microlens array 23 away from the pixel array 21. In this arrangement, the surface of the spatial light modulation array 22 itself has a protective function, thus eliminating the need for the cover glass on the surface of the traditional image sensor 20. Figure 5B As shown.

[0079] In this way, spatial light modulation arrays (such as spatial light polarization modulation (SLPM) arrays) can be integrated into existing image sensors to form new image sensors, which can eliminate the adverse effects of reflected light in the environment on imaging during image acquisition.

[0080] In some embodiments, the spatial light modulation array 22 is a spatial light polarization modulation (or also called a spatial light polarization modulation) array, which includes a plurality of light modulation units 220. For example... Figure 6A As shown, each optical modulation unit 220 includes a modulation subunit 221 and a polarizer 222, wherein the polarizer 222 is located on the side of the optical modulation unit 220 closer to the pixel array 21, and can be used as an analyzer. The modulation subunit 221 is a structural unit capable of adjusting the polarization state of light. Exemplarily, the modulation subunit 221 can be a liquid crystal-based modulation subunit or an electro-optic crystal-based modulation subunit, etc. Correspondingly, the optical modulation unit 220 can be a liquid crystal-based optical modulation unit or an electro-optic crystal-based optical modulation unit, etc.

[0081] In some embodiments, the modulation subunit 221 may be a liquid crystal-based modulation subunit, such as a liquid crystal unit, wherein the liquid crystal unit includes a nematic liquid crystal. In some embodiments, the liquid crystal unit includes a first electrode, a twisted-nematic unit, and a second electrode arranged sequentially, wherein the twisted-nematic unit includes a first alignment layer (alignment layer A), liquid crystal molecules, and a second alignment layer (alignment layer B). Further, the liquid crystal unit may be covered with glass on both the top and bottom.

[0082] For example, please refer to Figure 6B , Figure 6B This is a cross-sectional schematic diagram of a liquid crystal-based optical modulation unit 220 in one embodiment. (See diagram below.) Figure 6BAs shown, the optical modulation unit 220 includes a glass layer 223, an electrode layer 2211, a first alignment layer (alignment layer A) 2212, a nematic liquid crystal 2213, a second alignment layer (alignment layer B) 2214, and a polarizer 222. Specifically, light enters from above, and the pixel array of the image sensor 20 is shown on the lower side of this figure. The electrode layer 2211 is generally indium tin oxide (ITO) or other suitable materials that can act as conductors, coated on the glass unit. Alignment layers A 2212 and B 2214 are perpendicular to each other to form twisted-nematic units with nematic liquid crystal 2213 between them. The twisted-nematic effect is based on the controlled reorientation of nematic liquid crystals between different molecular configurations, which are based on the electric field (or voltage V) applied by the electrodes. Under the control of an external voltage, the liquid crystal 2213 can change the polarization direction of the incident light. Polarized light, after being processed by an electrically controlled liquid crystal, is selected by the bottom polarizer 222 (analyzer). This allows the polarization component perpendicular to the transmission direction of the polarizer 222 to be filtered out, thereby achieving the purpose of controlling the specific polarization component of the incident light. It should be noted that in the optical modulation unit 220 of this application, the polarizer 222 is arranged on only one side, and this polarizer 222 is located on the side closer to the pixel array 21, serving as an analyzer.

[0083] In some embodiments, the optical modulation unit 220 may also be made of other materials. Exemplarily, the optical modulation unit 220 may include an electro-optic crystal (e.g., lithium niobate) and a polarizer. Similarly, the polarizer functions as an analyzer and is positioned close to the pixel array 21. Faster and more stable modulation can be achieved using an electro-optic crystal.

[0084] It is understood that the liquid crystal unit / electro-optic crystal in the light modulation unit 220 is only a schematic example. With the development of technology, other feasible materials can be used to replace the liquid crystal unit / electro-optic crystal, as long as they can achieve the modulation of the polarization state of light.

[0085] In the above embodiments, the optical modulation unit can adjust the polarization state of the incident light under the drive of different control parameters (such as control voltage). By selecting a polarizer, the polarization component in the direction perpendicular to the transmission direction of the polarizer can be filtered out, thereby achieving the purpose of controlling a specific polarization component of the incident light.

[0086] In one exemplary embodiment, please refer to Figure 7This application provides an imaging control system 10, including an image sensor 20 and a processor 30. The image sensor 20 includes a pixel array 21 and a spatial light modulation array 22. The pixel array 21 includes a plurality of pixels 210; the spatial light modulation array 22 is located on the photosensitive side of the pixel array 21 and includes a plurality of light modulation units 220, each light modulation unit 220 being correspondingly disposed with at least one pixel 210. The processor 30 is connected to the image sensor 20 and is configured to independently adjust the control parameters of each light modulation unit 220 in the spatial light modulation array 22, so that the adjusted light modulation units 220 attenuate the polarized light in the incident light.

[0087] In the imaging control system of this application, the processor 30 is connected to the image sensor 20 and communicates with it via an interface (or bus). In some embodiments, the processor 30 is directly or indirectly connected to the pixel array 21 of the image sensor 20 via a first interface, and directly or indirectly connected to the spatial light modulation array 22 of the image sensor 20 via (at least one) second interface. In some embodiments, the processor 20 is directly or indirectly connected to each light modulation unit 220 via (at least one) second interface. The first interface (also called the data readout interface) is used to transmit image data acquired by the pixel array of the image sensor 20 to the processor 30, and the second interface (also called the control write interface) is used to transmit control signals from the processor 30 to each light modulation unit 220 of the spatial light modulation array 22. These control signals are used to control the electric field / voltage of the corresponding light modulation unit 220 (e.g., to control the control voltage written to each light modulation unit 220 (i.e., the control parameters mentioned above)).

[0088] In some embodiments, for any optical modulation unit 220, the specific process of the processor 30 adjusting the control parameters of the optical modulation unit 220 may include: the processor 30 sending a control signal to the optical modulation unit 220 through a second interface connected to the optical modulation unit 220, and using the digital-to-analog converter circuit (DAC circuit) at this interface to convert the control signal into a corresponding control voltage, so as to apply the corresponding control voltage to the optical modulation unit.

[0089] In some embodiments, the processor 30 may include, but is not limited to, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or even special dedicated image processing hardware (such as an ISP (Image Signal Processor)).

[0090] The processor 30 in this application can control the driving voltage (also called control voltage) applied to each optical modulation unit 220 in the spatial light modulation array 22 by adjusting the control parameters of each optical modulation unit 220, so that the adjusted optical modulation unit 220 can attenuate the locally polarized light in the incident light. It can be understood that the processor 30 can control each optical modulation unit 220 individually, thus enabling adaptive attenuation of polarized light in various directions and positions in the incident light. This allows polarization components in directions inconsistent with the polarizer's transmission direction to be filtered out, thereby achieving the purpose of controlling specific polarization components of the incident light.

[0091] The aforementioned imaging control system arranges a spatial light modulation array (particularly a spatial light polarization modulator) on the photosensitive side of the pixel array of the image sensor to modulate the polarization state of the incident light. This spatial light modulation array includes multiple light modulation units, each corresponding to at least one of the multiple pixels and independently controllable. The processor independently controls the control parameters of the multiple light modulation units in the spatial light modulation array, enabling adaptive attenuation of locally polarized light. This allows the light signal entering the image sensor to eliminate energy in the main polarization direction at different locations as much as possible, thereby weakening or even eliminating reflected light while simultaneously reducing or eliminating the object's information obscured by the originally reflected light.

[0092] In some embodiments, the light modulation unit 220 in the image sensor 20 of the imaging control system 10 includes a modulation subunit 221 and a polarizer 222, wherein the modulation subunit 221 is connected to the processor 300, and the polarizer 222 is located on the side of the light modulation unit 220 closer to the pixel array 21. In some embodiments, the modulation subunit 221 includes a liquid crystal-based modulation subunit or an electro-optic crystal-based modulation subunit.

[0093] In some embodiments, the image sensor 20 further includes a microlens array 23; wherein the microlens array 23 is located on the side of the spatial light modulation array 22 away from the pixel array 21, or the microlens array 23 is located between the spatial light modulation array 22 and the pixel array 21.

[0094] For a description of the image sensor, please refer to the foregoing embodiments; this embodiment will not repeat the description here.

[0095] The following section uses a liquid crystal-based optical modulation unit as an example to explain in detail the specific process of imaging control using the processor 30:

[0096] In some embodiments, the specific process of imaging control using the processor 30 may include a pre-shooting process and an actual shooting process. The purpose of the pre-shooting process is to analyze the polarization state at different locations in the scene to find the "optimal" control parameters applicable to each optical modulation unit 220. These "optimal" control parameters can control the corresponding optical modulation unit 220 to modulate its polarization transmission direction to be perpendicular to the direction of the locally polarized light incident upon it, thereby reducing or filtering out such polarized light. Thus, during the actual shooting process, image data can be acquired by the image sensor 20 after controlling the optical modulation unit 220 using the "optimal" control parameters, thereby obtaining a high-quality image that eliminates the influence of polarized light from different directions and locations.

[0097] Specifically, such as Figure 8 As shown, processor 30 can also be configured to perform steps 802 to 806.

[0098] Step 802: Change the control parameters of each optical modulation unit multiple times and use an image sensor to take a pre-shot.

[0099] The control parameters are used to control the optical modulation unit 220, specifically the control voltage (also called the driving voltage), which is the voltage applied to the optical modulation unit 220. When a certain control voltage is applied to the optical modulation unit 220, the optical modulation unit 220 can change the polarization direction of the incident light incident into it. For example, for a liquid crystal-based optical modulation unit 220, after applying a certain control voltage, the liquid crystal inside will rotate, thereby causing the polarization direction of the incident light incident into it to rotate synchronously.

[0100] For example, for each optical modulation unit 220, the processor 30 can successively control the electric field / voltage of the optical modulation unit 220 to different control voltages through an interface (such as a control write interface), and acquire image data using the image sensor 20 after each control is completed. In this embodiment, the image data acquired in this stage can be referred to as the pre-captured image.

[0101] In some embodiments, the control parameters of the optical modulation unit 220 are used to control the deflection angle corresponding to the optical modulation unit 220 (the deflection angle of the optical modulation unit 220 affects the polarization direction of the incident light), that is, there is a one-to-one correspondence between the control parameters of the optical modulation unit 220 and the deflection angle corresponding to the optical modulation unit 220. The deflection angle of the optical modulation unit 220 corresponds to the deflection angle of the polarization direction of the incident light. For example, if the optical modulation unit 220 is a liquid crystal-based optical modulation unit 220, the liquid crystal within the optical modulation unit 220 will rotate at a certain angle under the control of a certain control voltage. Correspondingly, the polarization direction of the incident light passing through the optical modulation unit 220 will also rotate synchronously with the rotation of the liquid crystal, where the rotation angle is the deflection angle.

[0102] During pre-shooting, multiple deflection angles can be sampled, and theoretically, equal spacing is not required for the sampling method. For convenience, uniform sampling can be considered. For example, [0, π / 2] can be uniformly sampled into N+1 angles to obtain N+1 deflection angles (where N is a positive integer greater than 1). That is, sampling can be performed every π / 2N radians starting from 0, up to π / 2. It can be understood that with the fixed structure of the liquid crystal-based light modulation unit 220, the correspondence between the deflection angle and the control parameter (control voltage) remains unchanged. That is, if the deflection angles obtained by the above sampling are determined, the control voltage corresponding to each deflection angle can be determined according to this correspondence.

[0103] Furthermore, the processor 30 can control each optical modulation unit 220 to traverse these control voltages for scene capture. Although theoretically, different optical modulation units 220 do not need to be under the same control voltage simultaneously during pre-capture, it is sufficient that each optical modulation unit 220 traverses the control voltage corresponding to the aforementioned N+1 deflection angles in all pre-captures. However, for ease of control, all optical modulation units 220 can be under the same control voltage (i.e., the polarization direction of all incident light is rotated by the same angle) during each pre-capture and then captured. In this embodiment, the pre-captured image obtained in each pre-capture can be denoted as... , where N is a positive integer greater than 1.

[0104] Step 804: Polarization analysis is performed based on the pre-captured image to determine the target control parameters of each optical modulation unit; the optical modulation unit controlled by the target control parameters is used to attenuate the polarized light in the incident light.

[0105] The processor 30 can acquire pre-captured images from the image sensor 20 via an interface (such as a data readout interface), and then perform scene polarization analysis on these pre-captured images to obtain the "optimal" control parameters (i.e., target control parameters) suitable for this scene. These "optimal" control parameters are then applied to the actual shooting in this scene. It should be noted that the "optimal" control parameters here consider eliminating or reducing polarized light in the scene. Reducing this polarized light, usually caused by specular reflection, can typically reduce the dynamic range of the scene, thereby increasing detail and restoring color at the original reflected light locations.

[0106] In some embodiments, the processor 30 is further configured to: perform polarization analysis based on the pre-captured image and determine the target control parameters of each optical modulation unit 220 by combining the correspondence between the optical modulation unit 220 and the pixel 210. The method for obtaining the correspondence between the optical modulation unit 220 and the pixel 210 is described in the following embodiments.

[0107] Step 806: Control the corresponding light modulation unit through the target control parameters and use the image sensor to take actual pictures.

[0108] Specifically, the processor 30 can determine the target control parameters adapted to each optical modulation unit 220, and then control the corresponding optical modulation unit 220 according to the target control parameters through an interface. For example, the target control parameters may specifically be a target control voltage, and the processor 30 can apply the corresponding target control voltage to each optical modulation unit 220 through the interface. After control is completed, actual imaging can be performed using the image sensor 20; that is, the image captured by the image sensor 20 at this time is the image that is truly put into application, such as being output and displayed as a target image.

[0109] For example, please refer to Figure 9 , Figure 9 This is a schematic flowchart illustrating the pre-shooting process in the imaging process according to an embodiment of this application. Figure 9 As shown, each optical modulation unit 220 is controlled to traverse the ITO voltage to pre-capture the scene and obtain a pre-captured image. Based on the set of pre-captured images, scene polarization analysis is performed, and the control parameters of the optical modulation unit 220 are analyzed in conjunction with the correspondence between the optical modulation unit 220 and the pixel 210, thereby outputting the target control parameters of the optical modulation unit 220. These target control parameters are used for actual shooting.

[0110] In the above embodiments, pre-shooting is performed by repeatedly changing the control reference of each optical modulation unit. Based on the analysis of the pre-shot images, the optimal control parameters (target control parameters) suitable for each optical modulation unit are found, and then the optimal control parameters are applied to the actual shooting. Since the optimal control parameters applied in the actual shooting process are those that help attenuate (or even eliminate) reflected light, the information that was originally obscured by reflected light can be recovered and presented in the image, greatly improving the image capture quality.

[0111] The process of determining the target control parameters will be explained in detail below:

[0112] In some embodiments, as described above, the control parameters of the optical modulation unit 220 are used to control the deflection angle corresponding to the corresponding optical modulation unit 220 (the deflection angle of the optical modulation unit 220 affects the polarization direction of the incident light), that is, there is a one-to-one correspondence between the control parameters of the optical modulation unit 220 and the deflection angle corresponding to the optical modulation unit 220.

[0113] Then, further, refer to Figure 10 The processor 30 can be configured to specifically execute steps 8042 to 8046 when executing step 804.

[0114] Step 8042: Based on the pixel information corresponding to the same pixel in multiple pre-captured images, determine the first deflection angle applicable to that pixel.

[0115] Specifically, the pixel information can be the color information (such as color values) and / or brightness information (such as brightness values) of the image pixels (i.e., pixels that correspond to pixels in the image sensor). In some embodiments, the pixel information of an image pixel can be the pixel value of that image pixel, which reflects the brightness or color intensity of the image at that image pixel.

[0116] The deflection angle is the angle by which the polarization direction of the incident light rotates. In the scenario of the liquid crystal-based light modulation unit 220, the deflection angle can also be understood as the rotation angle of the liquid crystal. It is affected by the control parameters and therefore has a corresponding relationship with the control parameters. The first deflection angle applicable to the pixel 210 satisfies the following condition: after the polarization direction of the polarized light in the incident light incident on the pixel 210 is rotated by the first deflection angle, it can be weakened or even eliminated (that is, it can attenuate the partially polarized light in the incident light incident on the pixel 210).

[0117] In some embodiments, for any pixel 210, the processor 30 can determine a first deflection angle that is compatible with it by means of the following: the processor 30 can filter out the pixel information of the image pixels that correspond to the pixel 210 in multiple pre-captured images, determine the deflection angle that minimizes the pixel information, and take the deflection angle as the first deflection angle.

[0118] In some embodiments, the processor 30 can directly use the deflection angle corresponding to the smallest pixel value in the pixel information as the first deflection angle. It is understood that during pre-capture, the control parameters corresponding to each light modulation unit 220 can be recorded, thus revealing the deflection angle corresponding to each light modulation unit 220. Therefore, based on the pre-captured image from which the smallest pixel value originates, the deflection angle corresponding to the light modulation unit 220 that corresponds to that pixel 210 can be determined; this deflection angle is the first deflection angle.

[0119] In some embodiments, due to the limitation of the number of samplings, the minimum value among the pixel information of a pixel determined based on multiple pre-captured images is not necessarily the theoretical minimum value of the pixel information. Therefore, the processor 30 is further configured to: for any pixel, perform function fitting based on the pixel information corresponding to that pixel in multiple pre-captured images, and determine a first deflection angle that minimizes the pixel information (such as pixel value) corresponding to that pixel based on the fitted function.

[0120] Specifically, in some embodiments, the processor 30 is further configured to: for any pixel, perform function fitting based on pixel information corresponding to the pixel in multiple pre-captured images, determine the deflection angle that maximizes the pixel information (such as pixel value) corresponding to the pixel according to the fitted function, and determine the first deflection angle that minimizes the pixel information (such as pixel value) corresponding to the pixel by utilizing the periodicity of the fitted function and according to the deflection angle that maximizes the pixel information (such as pixel value) corresponding to the pixel.

[0121] like Figure 11 As shown, the polarization direction allowed to pass through the polarizer surface of the light-collecting modulation unit 220 is the x-axis, the normal to the surface of the light-collecting modulation unit 220 is downward as the z-axis, and the direction perpendicular to the paper and outward is the y-direction. Assume that the vibration of the electric field of the incident light is represented as a vector within the plane of the paper. When no voltage is applied to the liquid crystal of the optical modulation unit 220, this electric field vibration is received by the analyzer. If a certain voltage is applied to the liquid crystal, the electric field vibration of the incident light will rotate by an angle θ. The vibration of the incident light before it reaches the analyzer and is received is: After being received by the analyzer, it becomes Considering By the Cauchy-Schwarz inequality, we have If and only if The equality holds true when the aforementioned conditions are met. That is, the intensity transmitted through the analyzer is at its maximum when the aforementioned conditions are satisfied. The objective of this application is to minimize the intensity transmitted through the analyzer, i.e., to ensure that the liquid crystal rotates at an angle perpendicular to the aforementioned angle.

[0122] Based on this principle, the process of scene polarization analysis is as follows. It should be noted that the coordinate system specified above is only for deriving the expression for the electric field intensity after polarization direction deflection and its maximum and minimum value properties. In actual use, other coordinate systems can be used, and the definition of the above coordinate system does not affect the scheme of this application.

[0123] During the aforementioned pre-shooting process, N+1 pre-shooting images were acquired under different control parameters (corresponding to different polarization angles). The deflection angles corresponding to the polarization directions are respectively .

[0124] For one of the image pixels At the deflection angle , The corresponding pixel value is below First, select the deflection index corresponding to the smallest pixel value. Considering that the pre-shooting process cannot be too long, otherwise it will be very unfriendly to dynamic scenes, the deflection angle sampling during pre-shooting will not be very dense, that is, N will not be very large. Thus, the pixel values ​​corresponding to the aforementioned deflection indices... It is also only optimal within the pre-shooting phase.

[0125] Therefore, this application can further optimize this angle. Analogous to subpixel interpolation in computer vision, this application can obtain the optimal deflection angle by employing subsampling interpolation. A most intuitive choice is to use a quadratic interpolation method, such as fitting a function using multiple pixel values, including the minimum pixel value of the pixel (e.g., pixel values ​​sampled before the minimum pixel value, the minimum pixel value, and pixel values ​​sampled after the minimum pixel value). For example, a deflection index can be used. The corresponding relationship is determined by fitting a quadratic function within this interval, thereby identifying the location of the minimum value of this quadratic function.

[0126] However, for functions of θ (The value of this function is proportional to the pixel value, thus reflecting the relationship between the pixel value and θ.) Since absolute value operations would destroy the smoothness and differentiability of this function near its zero point, this sub-pixel interpolation method utilizing the smoothness and extremum properties of quadratic functions is not suitable here. Let's change our perspective... It is a periodic function, and within one period, the difference between the positions of the maximum and minimum values ​​is... Furthermore, this function is smooth and differentiable near its maximum. Therefore, we can change our approach and first solve for the deflection angle corresponding to the maximum, and then obtain the deflection angle corresponding to the minimum based on the position difference.

[0127] Specifically, firstly, the deflection index corresponding to the maximum pixel value is selected. Then, a function is fitted using multiple pixel values, including the maximum pixel value of that pixel (such as pixel values ​​sampled before the maximum pixel value, the maximum pixel value, and pixel values ​​sampled after the maximum pixel value), for example, using a deflection index. The correspondence can be solved by fitting a quadratic function to find the maximum value where the maximum value appears. Of course, for a quadratic function to be suitable for fitting, the three points before and after must not form a linear relationship, meaning the denominator of the above equation must not be zero. This allows the pixel value to be calculated. The first deflection angle corresponding to the minimum value of the corresponding pixel:

[0128]

[0129] Thus, we obtained the “optimal” deflection angle that fits each pixel.

[0130] In the above embodiments, the "optimal" deflection angle applicable to each pixel can be accurately determined by function fitting.

[0131] Step 8044: For any optical modulation unit, determine the second deflection angle corresponding to the optical modulation unit based on the first deflection angle applicable to the pixel that has a corresponding relationship with the optical modulation unit.

[0132] In some embodiments, the processor 30 is further configured to: for any optical modulation unit, determine at least one pixel corresponding to the optical modulation unit based on the correspondence between the optical modulation unit and the pixel; and fuse the first deflection angle corresponding to the determined at least one pixel to obtain a second deflection angle corresponding to the optical modulation unit.

[0133] As previously mentioned, the spatial light modulation array 22 can physically cover the outside of the pixel array 21 of the image sensor 20 or further cover the outside of the microlens array 23. If the size of the spatial light modulation array 22 is relatively standardized and the installation is relatively standard, then the correspondence between the light modulation units and the pixels can be directly obtained based on the size and installation method of each light modulation unit 220 in the spatial light modulation array 22. Otherwise, in some other embodiments, the correspondence between the light modulation units and the pixels can be obtained in advance through calibration. Please refer to the following description of the embodiments for the calibration process.

[0134] It is understandable that, since the solution specifically adjusts the control parameters of each optical modulation unit 220 during execution, that is, controls the voltage applied to each optical modulation unit 220, and not necessarily each pixel 210 has an independently controlled voltage (control parameter), it is necessary to further determine the "optimal" deflection angle corresponding to each optical modulation unit 220.

[0135] For example, based on the correspondence between the light modulation unit and the pixel, for each light modulation unit pS i It can be based on all its corresponding pixel pI j First deflection angle By merging the components, a comprehensive deflection angle is obtained. This is the second deflection angle. It's conceivable that any fusion method is feasible. For example, one of the most intuitive approaches is to use the optical modulation unit pS... i The first deflection angle corresponding to the covered pixels The average value is taken to obtain the optical modulation unit pS. i The required second deflection angle Alternatively, select the optical modulation unit pS. i The first deflection angle corresponding to the covered pixels The median value of pS is used as the optical modulation unit. i The required second deflection angle However, the embodiments in this application do not limit this.

[0136] The second deflection angle corresponding to all optical modulation units In fact, it forms a two-dimensional deflection angle distribution map, and the processor 30 can output the corresponding voltage according to this distribution map to control the corresponding optical modulation unit.

[0137] In the above embodiments, after determining the deflection angle corresponding to each pixel, the deflection angles of all pixels associated with the optical modulation unit are fused to obtain the deflection angle applicable to the optical modulation unit.

[0138] Step 8046: Determine the target control parameters of the corresponding optical modulation unit based on the second deflection angle of each optical modulation unit.

[0139] Specifically, for each optical modulation unit 220, the processor 30 can calculate the specific value of the control parameter (specific control voltage value) that causes the optical modulation unit 220 to produce the corresponding deflection based on the second deflection angle corresponding to the optical modulation unit 220, thereby realizing the control of the optical modulation unit 220.

[0140] In other embodiments, the processor 30 is further configured to: smooth the second deflection angles corresponding to all optical modulation units in the spatial light modulation array to obtain a third deflection angle corresponding to each optical modulation unit; and determine the target control parameters of the corresponding optical modulation unit based on the third deflection angle corresponding to each optical modulation unit.

[0141] Considering that in practical processing, if the deflection angles of adjacent optical modulation units have significant jumps, it may cause the originally gradually changing region to be rotated significantly by different angles, resulting in various defects in the output image. Therefore, a post-processing procedure for the deflection angle can be optionally added. This procedure involves smoothing the second deflection angle of the optical modulation unit. This can be done using ordinary Gaussian smoothing, bilateral filtering, or even joint bilateral filtering guided by image content. This yields the smoothed third deflection angle of each optical modulation unit. Based on the third deflection angle of each optical modulation unit, the final target control parameters (such as the target control voltage) of the corresponding optical modulation unit can then be determined.

[0142] Thus, by determining the target control parameters of each optical modulation unit based on the deflection angle obtained after smoothing, obvious discontinuities and inconsistencies in the acquired images can be avoided, thereby improving the imaging quality.

[0143] During actual shooting, the target control parameters (target control voltage) output above for each optical modulation unit are used to drive each optical modulation unit to rotate the polarization direction of the incident polarized light to the optimal polarization direction desired by the scene, so as to reduce or remove the polarization component brought by the reflected light in the light signal as much as possible, thereby enabling the imaging result to better reflect the information of the scene itself.

[0144] The following section will provide a detailed explanation of the construction of the correspondence between the optical modulation unit and the pixel, which can also be referred to as the pre-calibration process.

[0145] In some embodiments, the processor 30 is further configured to: for each optical modulation unit, change the control parameters of the optical modulation unit while keeping the control parameters of other optical modulation units unchanged, and obtain multiple images of the optical modulation unit under different control parameters; for each optical modulation unit, based on the pixel information corresponding to the same pixel in the obtained multiple images, filter out the pixels affected by the optical modulation unit, and associate the pixels affected by the optical modulation unit with the optical modulation unit; and obtain the correspondence between the optical modulation unit and the pixel according to the pixels associated with each optical modulation unit in the spatial optical modulation array.

[0146] The correspondence between optical modulation units (PS) and pixels can be represented by a correspondence table. Throughout the calibration process, a correspondence table between optical modulation units (pS) and pixels (pI) is maintained. At the start of calibration, this correspondence table is initialized (cleared); at the end of calibration, the maintained correspondence table serves as the output of the calibration process.

[0147] During calibration, the entire device (including the image capture device with the aforementioned image sensor 20) is physically fixed and positioned, and the scene being captured is a fixed, static environment with polarized light characteristics. The exposure and gain parameters of the image sensor 20 are also at fixed values ​​during shooting.

[0148] refer to Figure 12A First, an outer loop is performed for each smallest controllable unit (also called an optical modulation unit 220) of the spatial optical modulation array 22. The following explanation uses a control voltage as the control parameter, assuming the currently traversed optical modulation unit is pS. i i=1,2,…NS (NS is the total number of optical modulation units in the spatial optical modulation array 22), first set all other optical modulation units pS j The control voltage, where j≠i, is a constant value. Based on this, all other optical modulation units pS... j It does not follow the inner loop of the optical modulation unit pS i The control voltage changes accordingly. The inner loop maintains a dataset of images acquired during the same inner loop process; this dataset is initialized (cleared) before the inner loop begins. Within the inner loop, for a fixed optical modulation unit pS... i It will traverse each control voltage within the preset voltage range and set the current optical modulation unit pS. i The control voltage is the currently traversed control voltage. Each time a different control voltage is set, a fixed scene is captured, and the captured image is added to the image dataset of the inner loop. Based on the characteristics of the spatial light modulation array 22, the control voltage of each of its aforementioned light modulation units can be... Variations within this range correspond to a deflection of the polarization direction of the incident light. Specifically, within the controllable voltage range, a series of voltage values ​​are sampled as traversal options, i.e., voltage... (Where NV is the total number of samples).

[0149] Since the purpose of calibration is only to find the correspondence between the light modulation unit and the pixel, and it is not necessary to perform strict calculations of the deflection angle, the value of voltage V does not need to be too strict.

[0150] For each optical modulation unit pS i The image dataset collected in the inner loop is Im. ik (x,y), i=1,2,…NS, k=1,2,…,NV. Where x,y are the pixel coordinate indices of the image sensor 20, i is the index of the corresponding optical modulation unit, and k represents the current image obtained by the optical modulation unit under control voltage V. k Images captured below.

[0151] Therefore, based on the pS of each optical modulation unit i The corresponding image dataset determines the optical modulation unit pS. i The affected pixels will be affected by the light modulation unit pS i The affected pixels and the light modulation unit pS i The association is then added to the correspondence table between the optical modulation unit and the pixel.

[0152] Specifically, for a given optical modulation unit pS i It can be used to count which pixels (x) in these images n ,y n The pixel values ​​(e.g., brightness values) corresponding to the given values ​​change significantly with k=1,2,…,NV. If the change is significant, then these pixels are generated by the light modulation unit pS. i The area covered by the pixel, that is, the area affected by the light modulation unit pS i The effect, i.e., pixel pI(x) n ,y n ) and optical modulation unit pS i There is a corresponding relationship.

[0153] In some embodiments, the processor 30 is further configured to: for each optical modulation unit, determine the pixel statistics of each pixel based on the pixel information corresponding to the same pixel in the multiple images obtained, and take the pixels whose pixel statistics satisfy a preset difference condition as the pixels affected by the optical modulation unit.

[0154] Pixel statistics are numerical values ​​obtained by statistically analyzing the pixel information of corresponding pixels in multiple images. For example, a pixel statistics value can be the difference between the maximum and minimum values ​​of pixel information corresponding to the same pixel in multiple images, or it can be the coefficient of variation calculated based on the pixel information of corresponding pixels in multiple images. Pixel statistics reflect the degree of variation of pixel information across different images. A preset difference condition is met for the pixel statistics value to be greater than or equal to a preset threshold.

[0155] It can be understood that when the structure of the imaging capture device is fixed, there is a fixed correspondence between the pixels of the image sensor 20 (or image sensor pixels) and the image pixels of the image acquired by the imaging capture device. This correspondence may or may not be one-to-one. Regardless of whether a one-to-one correspondence exists, the correspondence between the pixels of the image sensor 20 and the image pixels of the acquired image can be predetermined and remain unchanged when the structure of the image sensor 20 remains unchanged. Therefore, the pixel information of the image pixel corresponding to a certain pixel in the image can also be described as the pixel information corresponding to that pixel in the image.

[0156] For example, pixel statistics can be determined by calculating the standard deviation of pixel values ​​corresponding to the same pixel in multiple images. and mean If the ratio of the standard deviation to the mean (coefficient of variation) If the pixel statistics exceed a preset threshold (T0), the pixel statistics are considered to meet the preset difference condition, i.e.:

[0157]

[0158]

[0159]

[0160] All pixel pairs pS~pI with an Index value of 1 are recorded in the correspondence, while a value of 0 indicates that they are not recorded.

[0161] When the alignment between the optical modulation unit and the pixel is good and the size of the optical modulation unit is an integer multiple of the size of the pixel, the above process can ensure the accuracy of the correspondence between pS and pI.

[0162] However, if there is an error between the size of the light modulation unit 220 and the size of the pixel 210 of the image sensor 20, or if the installation position is not standardized, such that a light modulation unit pS does not exactly cover an integer number of pixels pI, the above correspondence can be optimized through a conflict checking and handling mechanism, so that each pixel in the correspondence has one and only one light modulation unit corresponding to it.

[0163] For example, the processor 30 is further configured to: if any pixel is associated with multiple light modulation units, select one light modulation unit from the multiple light modulation units associated with it as the final associated light modulation unit; if any pixel does not have an associated light modulation unit, select one light modulation unit to associate with the pixel based on the light modulation units associated with the pixels surrounding the pixel.

[0164] The specific process is as follows: Figure 12B As shown. Among them. Figure 12B Compared to Figure 12A The only difference is that before outputting the correspondence table between the light modulation unit and the pixel, a conflict check and an omission check are performed on the correspondence table between the light modulation unit and the pixel, and the correspondence table between the light modulation unit and the pixel is updated.

[0165] Specifically, to ensure that each pixel has one and only one corresponding optical modulation unit, when adding each candidate pS~pI correspondence, it is necessary to simultaneously record the pixel statistics (such as the ratio of standard deviation to mean) corresponding to that pixel. If there are multiple optical modulation units pS i All are related to a pixel pI(x) n ,y n If there is a corresponding match, then select the most reasonable one. For example, you could select the corresponding one. The largest optical modulation unit pS i The corresponding relationships are established, and the others are deleted. After resolving the above conflicts, for all pixels, there may still be some pixels, such as pI(x). m ,y m If (x) is not recorded in the corresponding relationship table, then it can be assigned according to the neighborhood principle. m ,y m The correspondence between other pixels in the vicinity is diffused to the current pixel, for example, including but not limited to the optical modulation unit pS that has been corresponded the most times in the neighboring pixels. p Assign to pixel pI(x) m ,y m And add the correspondence between the two to the correspondence table.

[0166] In the above embodiments, the correspondence between each optical modulation unit and each pixel can be accurately determined through the calibration process, and this correspondence is used in subsequent imaging control.

[0167] Based on the same inventive concept, this application also provides an imaging control method for an image sensor. The solution provided by this method is similar to the implementation scheme described in the imaging control system above. Therefore, the specific limitations in one or more imaging control method embodiments provided below can be found in the relevant limitations in the imaging control system above, and will not be repeated here.

[0168] An imaging control method for an image sensor, the image sensor including a pixel array having multiple pixels and a spatial light modulation array having multiple light modulation units, the spatial light modulation array being located on the photosensitive side of the pixel array, each light modulation unit being configured corresponding to at least one pixel; the method includes: independently adjusting the control parameters of each light modulation unit in the spatial light modulation array so that the adjusted light modulation unit attenuates the polarized light in the incident light.

[0169] In some embodiments, the control parameters of each optical modulation unit in the spatial light modulation array are independently adjusted so that the adjusted optical modulation unit attenuates the polarized light in the incident light, including: changing the control parameters of each optical modulation unit and taking a pre-shot using an image sensor; performing polarization analysis based on the pre-shot image to determine the target control parameters of each optical modulation unit; using the optical modulation unit controlled by the target control parameters to attenuate the polarized light in the incident light; and controlling the corresponding optical modulation unit through the target control parameters and taking an actual shot using an image sensor.

[0170] In some embodiments, the control parameters of the optical modulation unit are used to control the deflection angle of the corresponding optical modulation unit. Polarization analysis based on pre-captured images is used to determine the target control parameters of each optical modulation unit, including: determining a first deflection angle applicable to the pixel based on pixel information corresponding to the same pixel in multiple pre-captured images; for any optical modulation unit, determining a second deflection angle corresponding to the optical modulation unit based on the first deflection angle applicable to the pixel corresponding to the optical modulation unit; and determining the target control parameters of the corresponding optical modulation unit according to the second deflection angle corresponding to each optical modulation unit.

[0171] In some embodiments, determining a first deflection angle applicable to a pixel based on pixel information corresponding to the same pixel in multiple pre-captured images includes: for any pixel, performing function fitting based on pixel information corresponding to the pixel in multiple pre-captured images, and determining a first deflection angle that minimizes the pixel information corresponding to the pixel based on the fitted function.

[0172] In some embodiments, for any optical modulation unit, determining a second deflection angle corresponding to the optical modulation unit based on a first deflection angle applicable to a pixel that has a corresponding relationship with the optical modulation unit includes: for any optical modulation unit, determining at least one pixel corresponding to the optical modulation unit based on the correspondence between the optical modulation unit and the pixel; and fusing the first deflection angle corresponding to the determined at least one pixel to obtain the second deflection angle corresponding to the optical modulation unit.

[0173] In some embodiments, determining the target control parameters of a corresponding optical modulation unit based on the second deflection angle corresponding to each optical modulation unit includes: smoothing the second deflection angles corresponding to all optical modulation units in the spatial optical modulation array to obtain the third deflection angle corresponding to each optical modulation unit; and determining the target control parameters of the corresponding optical modulation unit based on the third deflection angle corresponding to each optical modulation unit.

[0174] In some embodiments, the method further includes: for each optical modulation unit, changing the control parameters of the optical modulation unit while keeping the control parameters of other optical modulation units unchanged, and obtaining multiple images of the optical modulation unit under different control parameters; for each optical modulation unit, filtering out pixels affected by the optical modulation unit based on pixel information corresponding to the same pixel in the obtained multiple images, and associating the pixels affected by the optical modulation unit with the optical modulation unit; and obtaining the correspondence between optical modulation units and pixels according to the pixels associated with each optical modulation unit in the spatial optical modulation array.

[0175] In some embodiments, for each optical modulation unit, based on the pixel information corresponding to the same pixel in multiple obtained images, the pixels affected by the optical modulation unit are selected, including: for each optical modulation unit, based on the pixel information corresponding to the same pixel in multiple obtained images, determining the pixel statistical value of each pixel, and selecting the pixels whose pixel statistical values ​​satisfy a preset difference condition as the pixels affected by the optical modulation unit.

[0176] In some embodiments, the method further includes: if any pixel is associated with multiple optical modulation units, then selecting one optical modulation unit from the multiple optical modulation units associated with it as the final associated optical modulation unit; if any pixel does not have an associated optical modulation unit, then selecting one optical modulation unit to associate with the pixel based on the optical modulation units associated with the pixels surrounding the pixel.

[0177] In one specific embodiment, the imaging control method mainly includes the following steps:

[0178] The spatial light modulation array is calibrated as follows: For each light modulation unit, while keeping the control parameters of other light modulation units constant, the control parameters of this light modulation unit are changed, and multiple images of this light modulation unit under different control parameters are obtained. For each light modulation unit, the pixel statistics of each pixel are determined based on the pixel information corresponding to the same pixel in the multiple obtained images. Pixels whose pixel statistics meet the preset difference conditions are identified as pixels affected by this light modulation unit, and the pixels affected by this light modulation unit are associated with this light modulation unit. The correspondence between light modulation units and pixels is obtained according to the pixels associated with each light modulation unit in the spatial light modulation array. If any pixel is associated with multiple light modulation units, one light modulation unit is selected from the multiple light modulation units associated with it as the final associated light modulation unit. If any pixel has no associated light modulation unit, one light modulation unit is selected and associated with the pixel based on the light modulation units associated with the pixels surrounding that pixel.

[0179] Pre-capture stage: Control parameters of each optical modulation unit are repeatedly changed, and pre-capture is performed using an image sensor. For any pixel, a function is fitted based on pixel information corresponding to that pixel in multiple pre-captured images, and a first deflection angle that minimizes the pixel information corresponding to that pixel is determined based on the fitted function. For any optical modulation unit, at least one pixel corresponding to that optical modulation unit is determined based on the correspondence between the optical modulation unit and pixels. The first deflection angle corresponding to the determined at least one pixel is fused to obtain a second deflection angle corresponding to that optical modulation unit. The second deflection angles corresponding to all optical modulation units in the spatial optical modulation array are smoothed to obtain a third deflection angle corresponding to each optical modulation unit. Based on the third deflection angle corresponding to each optical modulation unit, the target control parameters of the corresponding optical modulation unit are determined. The optical modulation unit controlled by the target control parameters is used to attenuate locally polarized light in the incident light.

[0180] Actual shooting stage: The corresponding light modulation unit is controlled by the target control parameters and the actual shooting is carried out using the image sensor.

[0181] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0182] Based on the same inventive concept, this application also provides an imaging control device for implementing the imaging control method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more imaging control device embodiments provided below can be found in the limitations of the imaging control system described above, and will not be repeated here.

[0183] In one exemplary embodiment, such as Figure 13 As shown, an imaging control device is provided, including: a pre-shooting module 1301, an analysis module 1302, and an actual shooting module 1303, wherein: the pre-shooting module 1301 is used to repeatedly change the control parameters of each of the optical modulation units and perform pre-shooting using the image sensor; the analysis module 1302 is used to perform polarization analysis based on the pre-shooting images to determine the target control parameters of each of the optical modulation units; the optical modulation units controlled by the target control parameters are used to attenuate the locally polarized light in the incident light; and the actual shooting module 1303 is used to control the corresponding optical modulation units through the target control parameters and perform actual shooting using the image sensor.

[0184] In some embodiments, the analysis module 1302 is further configured to determine a first deflection angle applicable to a pixel based on pixel information corresponding to the same pixel in multiple pre-captured images; for any optical modulation unit, determine a second deflection angle corresponding to the optical modulation unit based on the first deflection angle applicable to the pixel corresponding to the optical modulation unit; and determine the target control parameters of the corresponding optical modulation unit according to the second deflection angle corresponding to each optical modulation unit.

[0185] In some embodiments, the analysis module 1302 is further configured to perform function fitting on any pixel based on the pixel information corresponding to the pixel in multiple pre-captured images, and determine the first deflection angle that minimizes the pixel information corresponding to the pixel based on the fitted function.

[0186] In some embodiments, the analysis module 1302 is further configured to, for any optical modulation unit: determine at least one pixel corresponding to the optical modulation unit based on the correspondence between the optical modulation unit and the pixel; and fuse the first deflection angle corresponding to the determined at least one pixel to obtain a second deflection angle corresponding to the optical modulation unit.

[0187] In some embodiments, the analysis module 1302 is further configured to smooth the second deflection angles corresponding to all optical modulation units in the spatial optical modulation array to obtain the third deflection angle corresponding to each optical modulation unit; and to determine the target control parameters of the corresponding optical modulation unit based on the third deflection angle corresponding to each optical modulation unit.

[0188] In some embodiments, the device further includes a pre-calibration module, configured to, for each optical modulation unit, change the control parameters of the optical modulation unit while keeping the control parameters of other optical modulation units unchanged, and obtain multiple images of the optical modulation unit under different control parameters; for each optical modulation unit, based on the pixel information corresponding to the same pixel in the obtained multiple images, filter out the pixels affected by the optical modulation unit, and associate the pixels affected by the optical modulation unit with the optical modulation unit; and obtain the correspondence between the optical modulation unit and the pixel according to the pixels associated with each optical modulation unit in the spatial optical modulation array.

[0189] In some embodiments, the pre-calibration module is further configured to determine the pixel statistics of each pixel based on the pixel information corresponding to the same pixel in the multiple images obtained for each optical modulation unit, and to take the pixels whose pixel statistics satisfy the preset difference condition as the pixels affected by the optical modulation unit.

[0190] In some embodiments, the pre-calibration module is further configured to select one optical modulation unit from the plurality of optical modulation units associated with a pixel as the final associated optical modulation unit if any pixel is associated with a plurality of optical modulation units; and to select one optical modulation unit to associate with a pixel based on the optical modulation units associated with the pixels surrounding that pixel if any pixel is not associated with an optical modulation unit.

[0191] The modules in the aforementioned imaging control device can be implemented entirely or partially through software, hardware, or a combination thereof. In some cases, these modules can be embedded in the processor of a computer device in hardware form (such as circuits) or independent of the processor, or they can be stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to these modules.

[0192] In one embodiment, a processor is also provided, including logic circuitry capable of executing a computer program stored in a memory, wherein the logic circuitry performs the steps in the above-described method embodiments when executing the computer program.

[0193] In one embodiment, a non-volatile computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0194] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0195] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0196] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0197] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An imaging control system, characterized in that, The imaging control system includes: An image sensor, comprising a pixel array and a spatial light modulation array; wherein the pixel array comprises a plurality of pixels; the spatial light modulation array is located on the photosensitive side of the pixel array, and the spatial light modulation array comprises a plurality of light modulation units, each light modulation unit being disposed corresponding to at least one pixel; and The processor, connected to the image sensor, is configured to independently adjust the control parameters of each of the optical modulation units in the spatial light modulation array, so that the adjusted optical modulation units attenuate the polarized light in the incident light.

2. The imaging control system according to claim 1, characterized in that, The optical modulation unit includes: A modulation subunit, connected to the processor, and A polarizer is located on the side of the light modulation unit closer to the pixel array.

3. The imaging control system according to claim 1 or 2, characterized in that, The image sensor also includes: A microlens array; wherein the microlens array is located on the side of the spatial light modulation array away from the pixel array, or the microlens array is located between the spatial light modulation array and the pixel array.

4. The imaging control system according to claim 1, characterized in that, The processor is also configured to: The control parameters of each optical modulation unit are changed multiple times and pre-captured using the image sensor. Polarization analysis is performed based on the pre-captured image to determine the target control parameters of each of the optical modulation units; the optical modulation units controlled by the target control parameters are used to attenuate the polarized light in the incident light; The target control parameters are used to control the corresponding light modulation unit and the image sensor is used to take actual pictures.

5. The imaging control system according to claim 4, characterized in that, The control parameters of the optical modulation unit are used to control the deflection angle corresponding to the optical modulation unit, and the processor is further configured to: Based on the pixel information corresponding to the same pixel in multiple pre-captured images, determine the first deflection angle applicable to that pixel; For any optical modulation unit, a second deflection angle corresponding to the optical modulation unit is determined based on a first deflection angle applicable to a pixel that corresponds to the optical modulation unit. The target control parameters of each optical modulation unit are determined based on the second deflection angle corresponding to each optical modulation unit.

6. The imaging control system according to claim 5, characterized in that, The processor is also configured to: For any given pixel, a function is fitted based on the pixel information corresponding to that pixel in multiple pre-captured images, and the first deflection angle that minimizes the pixel information corresponding to that pixel is determined based on the fitted function.

7. The imaging control system according to claim 5, characterized in that, The processor is also configured to: For any optical modulation unit: Based on the correspondence between optical modulation units and pixels, at least one pixel corresponding to the optical modulation unit is determined; By fusing the first deflection angle corresponding to at least one pixel, a second deflection angle corresponding to the optical modulation unit is obtained.

8. The imaging control system according to claim 5, characterized in that, The processor is also configured to: The second deflection angles corresponding to all optical modulation units in the spatial light modulation array are smoothed to obtain the third deflection angle corresponding to each optical modulation unit. Based on the third deflection angle corresponding to each optical modulation unit, the target control parameters of the corresponding optical modulation unit are determined.

9. The imaging control system according to any one of claims 5 to 8, characterized in that, The processor is also configured to: For each optical modulation unit, while keeping the control parameters of other optical modulation units unchanged, the control parameters of this optical modulation unit are changed, and multiple images of this optical modulation unit under different control parameters are obtained. For each optical modulation unit, based on the pixel information corresponding to the same pixel in multiple obtained images, the pixels affected by the optical modulation unit are selected, and the pixels affected by the optical modulation unit are associated with the optical modulation unit. The correspondence between the optical modulation units and the pixels is obtained based on the pixels associated with each optical modulation unit in the spatial optical modulation array.

10. The imaging control system according to claim 9, characterized in that, The processor is also configured to: For each optical modulation unit, the pixel statistics of each pixel are determined based on the pixel information corresponding to the same pixel in the multiple images obtained, and the pixels whose pixel statistics meet the preset difference conditions are regarded as the pixels affected by the optical modulation unit.

11. The imaging control system according to claim 9, characterized in that, The processor is also configured to: If any pixel is associated with multiple optical modulation units, then select one optical modulation unit from the multiple optical modulation units associated with it as the final associated optical modulation unit; If any pixel does not have an associated optical modulation unit, then an optical modulation unit is selected and associated with that pixel based on the optical modulation units associated with the pixels surrounding that pixel.

12. An imaging control method for an image sensor, characterized in that, The image sensor includes a pixel array with multiple pixels and a spatial light modulation array with multiple light modulation units, the spatial light modulation array being located on the photosensitive side of the pixel array, and each light modulation unit being configured corresponding to at least one pixel; the method includes: The control parameters of each optical modulation unit in the spatial light modulation array are independently adjusted so that the adjusted optical modulation unit attenuates the polarized light in the incident light.

13. The method according to claim 12, characterized in that, The independent adjustment of the control parameters of each optical modulation unit in the spatial light modulation array, so that the adjusted optical modulation unit attenuates the polarized light in the incident light, includes: The control parameters of each optical modulation unit are changed multiple times and pre-captured using the image sensor. Polarization analysis is performed based on pre-captured images to determine the target control parameters for each of the optical modulation units; the optical modulation units controlled by the target control parameters are used to attenuate the polarized light in the incident light; The target control parameters are used to control the corresponding light modulation unit and the image sensor is used to take actual pictures.

14. An image sensor, characterized in that, include: A pixel array, the pixel array comprising a plurality of pixels; A spatial light modulation array is located on the photosensitive side of the pixel array. The spatial light modulation array includes multiple light modulation units, and each light modulation unit is configured corresponding to at least one pixel. The optical modulation unit can be independently controlled to modulate the polarization state of the incident light. The optical modulation unit attenuates the polarized light in the incident light to different degrees depending on the polarization state of the incident light.

15. The image sensor according to claim 14, characterized in that, The optical modulation unit includes: Modulation subunit, and A polarizer is located on the side of the light modulation unit closer to the pixel array.

16. The image sensor according to claim 15, characterized in that, The modulation subunit includes a liquid crystal-based modulation subunit or an electro-optic crystal-based modulation subunit.

17. The image sensor according to any one of claims 14 to 16, characterized in that, The image sensor also includes: A microlens array having multiple microlenses, each microlens corresponding to at least one pixel; wherein the microlens array is located on the side of the spatial light modulation array away from the pixel array, or the microlens array is located between the spatial light modulation array and the pixel array.

18. A processor comprising logic circuitry capable of executing a computer program stored in a memory, wherein the logic circuitry, when executing the computer program, implements the steps of the method of claim 12 or 13.

19. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 12 or 13.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 12 or 13.

Images