Camera module, electronic device, and photographing method and apparatus

Abstract

The present application belongs to the technical field of camera photographing. Disclosed are a camera module, an electronic device, a photographing method and apparatus, and a readable storage medium. The camera module comprises a first lens assembly, a total-reflection mirror, a dimming assembly, an image sensor and a feedback circuit, wherein the first lens assembly, the total-reflection mirror and the dimming assembly are sequentially arranged, and the image sensor and the total reflection mirror are arranged opposite each other; the image sensor comprises at least one photosensitive region, and the dimming assembly comprises at least one dimming region, one photosensitive region corresponding to one dimming region; the feedback circuit is connected to both the image sensor and the dimming assembly, and the feedback circuit is used for obtaining a feedback signal on the basis of image data collected by the first photosensitive region in the image sensor, and transmitting the feedback signal to the dimming assembly; and the dimming assembly is used for adjusting the reflectivity of a first dimming region, which corresponds to a first photosensitive region, in the dimming assembly on the basis of the feedback signal.

Description

Camera module, electronic device, photographing method and apparatus

[0001] Cross-reference to Related Applications

[0002] This application claims priority to the Chinese patent application No. 202410464340.3 filed on April 17, 2024 in China, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application belongs to the technical field of camera technology, and specifically relates to a camera module, an electronic device, a photographing method and an apparatus. BACKGROUND

[0004] Generally, the imaging performance of a camera device in a same scene for a high-light area and a dim area is limited. The high-light area may be overexposed and appear white after imaging, and the dim area may be underexposed and appear black after imaging. This limitation is a lack of dynamic range, which affects the image quality after imaging. High dynamic range imaging (HDR) technology is an imaging technology that can achieve a larger dynamic range.

[0005] To achieve high dynamic range imaging, a conventional device usually adopts a multi-exposure HDR technology, that is, multiple images are captured through different exposure strategies, and then the multiple images are fused into a high dynamic image. Because the high dynamic image after fusion is naturally exposed, it can avoid overexposure of the high-light area in the image and underexposure of the dim area in the image, thereby achieving the requirements of high dynamic range imaging. However, because there is a time difference when the image is captured by using the multi-exposure HDR technology, there may be a motion blur problem in the high dynamic image formed by fusing the multiple images, which affects the imaging quality of the camera device. SUMMARY

[0006] The present application aims to provide a camera module, an electronic device, a photographing method and an apparatus to solve the problem that there is a time difference when the image is captured by using the multi-exposure HDR technology in the related technology, which causes a motion blur problem in the high dynamic image formed by fusing the multiple images captured through different exposure strategies.

[0007] To solve the above technical problem, the present application is implemented as follows:

[0008] In a first aspect, embodiments of the present application provide a camera module, comprising a first lens assembly, a total reflection mirror, a light adjusting assembly, an image sensor, and a feedback circuit, the first lens assembly, the total reflection mirror, and the light adjusting assembly are sequentially arranged, and the image sensor is arranged opposite to the total reflection mirror; incident light is reflected by the total reflection mirror after being converged by the first lens assembly and entering the light adjusting assembly, the light adjusting assembly reflects the incident light after modulating the incident light, and the total reflection mirror reflects the incident light to the image sensor; the image sensor comprises at least one photosensitive region, the light adjusting assembly comprises at least one light adjusting region, and one photosensitive region corresponds to one light adjusting region; the feedback circuit is connected with the image sensor and the light adjusting assembly respectively; the feedback circuit is configured to obtain a feedback signal based on image data collected by a first photosensitive region in the image sensor and transmit the feedback signal to the light adjusting assembly; and the light adjusting assembly is configured to adjust reflectivity of a first light adjusting region corresponding to the first photosensitive region in the light adjusting assembly based on the feedback signal.

[0009] In a second aspect, embodiments of the present application provide an electronic device, comprising the camera module of the first aspect.

[0010] In a third aspect, embodiments of the present application provide an image processing method, executed by an electronic device comprising the camera module of the first aspect, the method comprising: controlling a feedback circuit of the camera module to obtain first image data collected by an image sensor in a first photosensitive region in the camera module; controlling the feedback circuit in the camera module to obtain a first feedback signal based on the first image data; controlling a light adjusting assembly in the camera module to adjust reflectivity of a first light adjusting region corresponding to the first photosensitive region in the light adjusting assembly based on the first feedback signal; and controlling the image sensor to collect second image data.

[0011] In a fourth aspect, embodiments of the present application provide an image processing apparatus, comprising the camera module of the first aspect; the image processing apparatus further comprises a processing module, configured to control a feedback circuit of the camera module to obtain first image data collected by an image sensor in a first photosensitive region in the camera module; the processing module is further configured to control the feedback circuit in the camera module to obtain a first feedback signal based on the first image data; the processing module is further configured to control a light adjusting assembly in the camera module to adjust reflectivity of a first light adjusting region corresponding to the first photosensitive region in the light adjusting assembly based on the first feedback signal; and the processing module is further configured to control the image sensor to collect second image data.

[0012] In a fifth aspect, embodiments of the present application provide an electronic device, comprising a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the method of the third aspect.

[0013] In a sixth aspect, an embodiment of the present application provides a readable storage medium, the readable storage medium storing a program or instructions, the program or instructions being executed by a processor to implement the steps of the method in the third aspect.

[0014] In a seventh aspect, an embodiment of the present application provides a chip, the chip comprising a processor and a communication interface, the communication interface being coupled to the processor, the processor being configured to run a program or instructions to implement the steps of the method in the third aspect.

[0015] In an eighth aspect, an embodiment of the present application provides a computer program product, the program product being stored in a storage medium, the program product being executed by at least one processor to implement the steps of the method in the third aspect.

[0016] In the embodiments of the present application, since the image sensor can collect image data of the environment where the camera module is located, the feedback circuit can obtain a feedback signal based on the image data collected by the first light sensing area in the image sensor; and the light adjusting assembly can adjust the reflectivity of the first light adjusting area corresponding to the first light sensing area in the light adjusting assembly based on the feedback signal. Therefore, the feedback circuit can obtain the feedback signal based on the brightness value of the first light sensing area and transmit the feedback signal to the light adjusting assembly, the light adjusting assembly can reduce or enhance the reflectivity of the first light adjusting area based on the feedback signal, so as to change the intensity of the light reflected via the first light adjusting area, so that the light reflected to the total reflector via the first light adjusting area of the light adjusting assembly, and then reflected to the first light sensing area of the image sensor, the image sensor can obtain an image with normal exposure, thereby avoiding overexposure of the first light sensing area due to too high light intensity, and further achieving the requirement of high dynamic range imaging and improving the imaging quality of the camera module. Moreover, when the camera module in the above embodiments of the present application is used, since the multiple exposure HDR technology is not used for imaging, the camera module does not need to fuse multiple images into one image when capturing an image, thereby avoiding the problem that the synthesized image is blurred due to the time difference between multiple images, and further improving the imaging quality of the camera module. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings in which:

[0018] FIG. 1 is a schematic diagram of a camera module according to an embodiment of the present application;

[0019] FIG. 2 is a schematic diagram of a pixel array and a light adjusting array according to an embodiment of the present application;

[0020] FIG. 3 is a schematic diagram of a light adjusting unit according to an embodiment of the present application;

[0021] FIG. 4 is a schematic diagram II of the light adjusting unit according to an embodiment of the present application;

[0022] FIG. 5 is a schematic diagram II of the camera module according to an embodiment of the present application;

[0023] FIG. 6 is a schematic diagram III of the camera module according to an embodiment of the present application;

[0024] FIG. 7 is a schematic diagram I of the image processing method according to an embodiment of the present application;

[0025] FIG. 8 is a schematic diagram II of the image processing method according to an embodiment of the present application;

[0026] FIG. 9 is a schematic diagram of the first function relationship according to an embodiment of the present application;

[0027] FIG. 10 is a schematic diagram III of the image processing method according to an embodiment of the present application;

[0028] FIG. 11 is a schematic diagram of the pixel array on the image sensor and the light adjusting array on the light adjusting component when using the chessboard projection method for calibration according to an embodiment of the present application;

[0029] FIG. 12 is a schematic diagram of the data formed by the encoding module according to an embodiment of the present application;

[0030] FIG. 13 is a schematic diagram IV of the image processing method according to an embodiment of the present application;

[0031] FIG. 14 is a schematic diagram of the image processing device according to an embodiment of the present application;

[0032] FIG. 15 is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present application;

[0033] FIG. 16 is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. The same or similar components are denoted by the same reference numerals throughout the drawings and a repeated description is omitted. The embodiments described below are merely examples for explaining the present application and should not be construed as limiting the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0035] The terms "first", "second", "third", "fourth" in the specification of the present application can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified. In addition, "and / or" in the specification means at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in a "or" relationship.

[0036] In the description of the present application, it should be understood that the orientation or positional relationship indicated by the terms "center", "axial", "radial", "circumferential", "upper", "inner", "outer", "long", "short", "two sides", "between" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0037] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected", "connected" should be broadly understood, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0038] The terms related to the embodiments of the present application will be described below.

[0039] 1、Camera Compact Module

[0040] Camera Compact Module (CCM) is an important tool for obtaining visual information, which is usually composed of Lens, IR Cut Filter, Image Sensor and Voice Coil Motor (VCM).

[0041] The working process of CCM is as follows: first, the voice coil motor drives the lens to focus, and the light carrying visual information is focused through the lens, and the infrared band invisible to the human eye is filtered out through the infrared cut filter, and the image is formed on the image sensor. Second, the photodiode in the image sensor converts the light signal into an electrical signal, and forms a digital signal matrix, i.e. RAW image, through amplification and analog-to-digital conversion. Finally, after processing by the Image Signal Process (ISP) module, it is compressed and stored in the memory. Among them, the RAW image is the original data of the digital signal converted by the image sensor from the captured light source signal.

[0042] 2、image sensor

[0043] An image sensor is a device that converts an optical image into an electronic signal, and is a core component of a camera module. Today, image sensors are mainly divided into charge coupled device (CCD) elements and complementary metal oxide semiconductor (CMOS) devices. The core function of an image sensor is to sense light, i.e., to convert a detected light signal into a digital signal that is convenient for transmission and processing.

[0044] 3、complementary metal oxide semiconductor image sensor

[0045] A complementary metal oxide semiconductor image sensor (CMOS Image Sensor, CIS) is a type of image sensor that uses a CMOS semiconductor. Compared with a CCD image sensor, a CMOS image sensor has lower cost and power consumption, and is more flexible in reading. The working mechanism of a CIS is as follows: a light sensing diode arranged in an array senses a light signal and converts it into an electrical signal, an amplification circuit and an analog-to-digital conversion (ADC) circuit are used to form a digital signal matrix, i.e., an image, and an ISP module is used to process the image data and compress and store the image data.

[0046] 4、lens

[0047] A lens is another core component of a camera module, and its function is to converge light rays in a field of view into an imaging plane. Its optical quality directly affects the imaging quality. From the perspective of information transmission, the design level of the lens determines the transmission of high-frequency information to the image plane, thereby directly affecting the imaging clarity of the module. From the perspective of energy transmission, the lens structure and transmittance determine the intensity and distribution of light reaching the CIS, thereby directly affecting the signal-to-noise ratio performance of the CIS.

[0048] 5、dynamic range

[0049] Dynamic range (DR) refers to the ability of an image sensor to simultaneously represent high light and shadow content in an image. The formula is: DR = 20*log(I max / I min ), where I max is the maximum unsaturated current of the image sensor, i.e., the current when the image sensor is just saturated, and I minBlack level refers to the bottom current of the image sensor. High dynamic range imaging (HDR) technology is an imaging technology that can achieve a larger dynamic range.

[0050] In addition, the dynamic range is also positioned as the ratio between the full well capacity (FWC) of the image sensor and the noise floor. The full well capacity refers to the maximum amount of charge that the capacitor of the photodiode of the image sensor can accumulate. The noise floor, also known as read noise, is defined as the noise generated by the readout circuit, which does not include the noise generated in the detector. In a CMOS image sensor, the readout circuit includes an amplifier inside the pixel. Therefore, by increasing the full well capacity or reducing the noise floor, the dynamic range of the image sensor can be improved, allowing the image sensor to meet the imaging requirements of high dynamic range imaging. It can be understood that when the dynamic range of the image sensor is small, the image obtained by the image sensor will have a high-light area or a dark area. The high-light area will have excessive exposure, resulting in excessive brightness in the area, and the dark area will have insufficient exposure, resulting in loss of details, affecting the quality of the image collected by the image sensor.

[0051] 6. Silicon-based liquid crystal chip

[0052] The silicon-based liquid crystal chip (Liquid Crystal on Silicon, LCoS) is an optical element that combines liquid crystal and silicon chip technology. It can input electrical signals to the semiconductor chip through a small interconnection lead, drive the liquid crystal array to change the pixel state, and thus change the intensity of reflected light.

[0053] In recent years, camera modules have become a standard module for terminal devices such as smartphones and tablets as an important tool for obtaining visual information and artistic creation. Due to the advantages of portability and powerful computing power of mobile terminals, consumers have increasingly high requirements for the imaging quality and shooting functions of mobile terminals. Among them, mobile terminals have increasingly high requirements for the dynamic range of camera modules. However, due to the limited internal space of mobile terminals, the size of the image sensor of the camera module in the mobile terminal is difficult to further increase, and the full well capacity of the CIS is greatly limited, resulting in the inability of the camera module to achieve high dynamic range imaging requirements, affecting the imaging quality of the camera module. Therefore, how to further improve the dynamic range of the camera module has become a problem that needs to be solved by those skilled in the art.

[0054] Generally, the imaging performance of a camera device in the same scene for high-light areas and dark areas is limited. The high-light area may appear white after imaging due to overexposure, and the dark area may appear black after imaging due to underexposure. This limitation is caused by insufficient dynamic range.

[0055] In order to realize high dynamic range imaging, a conventional device usually adopts a multi-exposure HDR technology, that is, multiple images are captured through different exposure strategies, and then the multiple images are fused into one high dynamic image. Because the high dynamic image after fusion has normal exposure, it can avoid overexposure in highlight areas and underexposure in dark areas in the image, so as to realize the requirements of high dynamic range imaging. However, because there is a time difference when the image is captured by using the multi-exposure HDR technology, there may be a motion blur problem in the high dynamic image formed by fusing multiple images, which affects the imaging quality of the camera device.

[0056] In addition, in order to realize high dynamic range imaging, the conventional device can also adopt HDR technologies based on dual conversion gain (DCG) and digital overlap readout (DOL), etc. The dual conversion gain technology is a technology of increasing two photoelectric conversion amplification circuits with different amplification powers, and the digital overlap technology is a technology of overlapping readout and fusion of images with different exposure times. The above technologies can realize the requirements of high dynamic range imaging, but increase the complexity and manufacturing cost of the image sensor circuit design.

[0057] In order to solve the above problems, an embodiment of the present application adopts a camera module, which includes a first lens assembly, a total reflection mirror, a light adjusting assembly, an image sensor, and a feedback circuit. The first lens assembly, the total reflection mirror, and the light adjusting assembly are sequentially arranged, and the image sensor is arranged opposite to the total reflection mirror. After the incident light is converged by the first lens assembly and enters the light adjusting assembly through the total reflection mirror, the light adjusting assembly reflects the modulated incident light to the total reflection mirror, and the total reflection mirror reflects the incident light to the image sensor. The image sensor includes at least one photosensitive area, the light adjusting assembly includes at least one light adjusting area, and one photosensitive area corresponds to one light adjusting area. The feedback circuit is connected with the image sensor and the light adjusting assembly respectively. The feedback circuit is used to obtain a feedback signal based on image data collected by a first photosensitive area in the image sensor and transmit the feedback signal to the light adjusting assembly. The light adjusting assembly is used to adjust the reflectivity of a first light adjusting area corresponding to the first photosensitive area in the light adjusting assembly based on the feedback signal.

[0058] Since the image sensor can collect image data of the environment where the camera module is located, the feedback circuit can obtain a feedback signal based on the image data collected by the first photosensitive region in the image sensor; the light adjustment assembly can adjust the reflectivity of the first light adjustment region corresponding to the first photosensitive region in the light adjustment assembly based on the feedback signal. Therefore, the feedback circuit can obtain a feedback signal based on the brightness value of the first photosensitive region and transmit it to the light adjustment assembly. The light adjustment assembly can reduce or enhance the reflectivity of the first light adjustment region based on the feedback signal, thereby changing the intensity of the light reflected via the first light adjustment region. The light is reflected to the total reflector through the first light adjustment region of the light adjustment assembly, and then reflected to the first photosensitive region of the image sensor through the total reflector. After that, the image sensor can obtain an image with normal exposure, thereby avoiding overexposure of the first photosensitive region due to excessive light intensity, and further achieving the requirement of high dynamic range imaging and improving the imaging quality of the camera module. Moreover, when the camera module of the present application is used, multiple exposure HDR technology is not required for imaging, so that the camera module does not need to fuse multiple images into one image when capturing an image, thereby avoiding the problem that the fused image is blurred due to the time difference between multiple images, and further improving the imaging quality of the camera module. Moreover, the circuit structure of the image sensor of the present application is simple and the manufacturing cost is low. At the same time, the camera module of the present application can be used with different types of image sensors. The high dynamic range function can be realized on the main camera, long focal, wide angle, front camera and other modules, and the HDR shooting and recording functions can be realized in all scenes, which is conducive to volume multiplexing and thus reduces the cost.

[0059] The camera module 100 provided by the embodiments of the present application will be described in detail below in combination with the drawings, specific embodiments and application scenarios. FIGS. 1 to 6 show possible structural schematic diagrams of a camera module 100 provided by the embodiments of the present application.

[0060] As shown in FIG. 1, the camera module 100 includes a first lens assembly 1, a total reflection mirror 2, a light adjusting assembly 3, an image sensor 4, and a feedback circuit 5. The first lens assembly 1, the total reflection mirror 2, and the light adjusting assembly 3 are sequentially arranged, and the image sensor 4 is arranged opposite to the total reflection mirror 2. After the incident light is converged by the first lens assembly 1 and enters the light adjusting assembly 3 through the total reflection mirror 2, the light adjusting assembly 3 reflects the modulated incident light to the total reflection mirror 2, and the total reflection mirror 2 reflects the incident light to the image sensor 4. The image sensor 4 includes at least one photosensitive region 41, the light adjusting assembly 3 includes at least one light adjusting region 31, and one photosensitive region 41 corresponds to one light adjusting region 31. The feedback circuit 5 is connected to the image sensor 4 and the light adjusting assembly 3, respectively. The feedback circuit 5 is configured to obtain a feedback signal based on image data collected by a first photosensitive region of the image sensor 4 and transmit the feedback signal to the light adjusting assembly 3. The light adjusting assembly 3 is configured to adjust the reflectivity of a first light adjusting region of the light adjusting assembly 3 corresponding to the first photosensitive region based on the feedback signal.

[0061] In some embodiments of the present application, the first lens assembly 1 is composed of multiple aspheric lenses, which can be equivalent to a convex lens and has the function of converging incident light, so that the light can be converged to the light adjusting assembly 3 through the first lens assembly 1 and the total reflection mirror 2.

[0062] In some embodiments of the present application, the light can be imaged for the first time after passing through the first lens assembly 1 and the total reflection mirror 2, that is, an intermediate image plane can be generated. The light adjusting assembly 3 can be arranged at the position of the intermediate image plane.

[0063] In some embodiments of the present application, the total reflection mirror 2, also known as an internal total reflection prism, is a prism manufactured based on the principle of total reflection. In the embodiments of the present application, the total reflection mirror 2 has the function of totally reflecting the light modulated by the light adjusting assembly 3, so that the light after total reflection enters the image sensor 4, avoids loss of light during reflection, and realizes lossless transmission of light.

[0064] In some embodiments of the present application, by controlling the proportional relationship between the incident angle of the incident light reflected by the total reflection mirror 2 and the exit angle of the exit light, it can be ensured that the light modulated by the light adjusting assembly 3 can be totally reflected to the image sensor 4 when passing through the total reflection mirror 2, avoiding loss of light during reflection. According to the principle and formula of total reflection, sinC = 1 / n, C is the total reflection angle, and n is the refractive index of the prism. It can be known that by controlling the incident angle of the incident light reflected by the total reflection mirror 2 to be greater than the total reflection angle C, it can be ensured that the light modulated by the light adjusting assembly 3 can be totally reflected to the image sensor 4 when passing through the total reflection mirror 2.

[0065] In some embodiments of the present application, as shown in FIG. 1, the total reflection mirror 2 includes oppositely arranged first mirror body 21 and second mirror body 22, the first mirror body 21 is arranged opposite to the first lens assembly 1, and the second mirror body 22 has a reflecting surface; the incident light rays passing through the first lens assembly 1 are transmitted to the light modulation assembly 3 in sequence through the first mirror body 21 and the second mirror body 22, reflected to the reflecting surface of the second mirror body 22 after being modulated by the light modulation assembly 3, and then reflected to the image sensor 4 through the reflecting surface.

[0066] It can be understood that when the incident light rays are transmitted to the first mirror body 21 through the first lens assembly 1, the first mirror body 21 can correct and converge the light rays to avoid deviation of the transmission direction of the incident light rays. When the incident light rays are reflected to the reflecting surface of the second mirror body 22 after being modulated by the light modulation assembly 3, the incident light rays can be totally reflected and reflected to the image sensor 4, so that the image sensor 4 can collect the light information of the environment where the camera module 100 is located, and realize lossless transmission of the light rays.

[0067] In some embodiments of the present application, a transmission film can be coated on the first mirror body 21 to improve the transmittance of the first mirror body 21 and reduce the loss of the incident light rays when transmitted through the first mirror body 21.

[0068] In some embodiments of the present application, the camera module 100 further includes a driving device for driving the light modulation assembly 3 to move, so as to adjust the included angle between the exit surface of the light modulation assembly 3 and the optical axis of the first lens assembly 1.

[0069] It can be understood that by driving the light modulation assembly 3 to move through the driving device, the included angle between the exit surface of the light modulation assembly 3 and the optical axis of the first lens assembly 1 is adjusted, so that the incident angle of the incident light rays reflected by the total reflection mirror 2 can be adjusted. When the incident light rays modulated by the light modulation assembly 3 are emitted from the exit surface of the light modulation assembly 3 and irradiate to the reflecting surface of the second mirror body 22, it can be ensured that the incident angle of the incident light rays reflected by the total reflection mirror 2 is greater than the total reflection angle, so that the light rays modulated by the light modulation assembly 3 can be totally reflected to the image sensor 4 when passing through the total reflection mirror 2, and lossless transmission of the light rays is realized.

[0070] In some embodiments of the present application, the driving device can be a linear actuator, such as a gear and rack mechanism, a screw and nut mechanism, etc.; the driving device can also be a rotating mechanism, etc. The embodiments of the present application are not limited here.

[0071] In some embodiments of the present application, the driving device can drive the light adjustment assembly 3 to move in a direction perpendicular to the optical axis of the first lens assembly 1, so as to adjust the angle of inclination of the light adjustment assembly 3, and then adjust the angle between the exit surface of the light adjustment assembly 3 and the optical axis of the first lens assembly 1, that is, the angle between the exit surface of the light adjustment assembly 3 and the reflecting surface of the second mirror 22.

[0072] In some embodiments of the present application, the light adjustment assembly 3 includes two or more light adjustment regions 31, and different light adjustment regions 31 can adjust incident light rays of different incident angles, respectively.

[0073] In some embodiments of the present application, the image sensor 4 includes two or more light sensing regions 41, and one light sensing region 41 corresponds to one light adjustment region 31. Different light adjustment regions 31 can individually adjust the brightness value of the corresponding light sensing region 41, so that the light adjustment assembly 3 can accurately adjust the brightness value of a single pixel point in the image collected by the image sensor 4, avoid overexposure caused by excessive light intensity in some areas of the image, and then achieve the requirement of high dynamic range imaging, and improve the imaging quality of the camera module 100.

[0074] In some embodiments of the present application, the image data collected by the first light sensing region in the image sensor 4 can be the brightness value, position information, etc. of the first light sensing region.

[0075] In some embodiments of the present application, as shown in FIG. 2, the image sensor 4 includes a pixel array, and the pixel array includes at least one pixel unit 8. The light adjustment assembly 3 includes a light adjustment array, and the light adjustment array includes at least one light adjustment unit 9. Each pixel unit 8 in the pixel array corresponds to one light adjustment unit 9 in the light adjustment array.

[0076] It can be understood that the feedback circuit 5 can generate a feedback signal based on the image data collected by the target pixel unit 8 in the image sensor 4, and transmit the feedback signal to the light adjustment assembly 3. The light adjustment assembly 3 can adjust the reflectivity of the target light adjustment unit 9 in the light adjustment assembly 3 based on the feedback signal, so as to adjust the intensity of the light reflected by the target light adjustment unit 9. Then, when the light reflected by the target light adjustment unit 9 is reflected into the target pixel unit 8 by the total reflection mirror 2, the excessive exposure caused by excessive light intensity can be avoided, so that the light adjustment assembly 3 can accurately adjust the brightness value of a single pixel point in the image collected by the image sensor 4, and then the camera module 100 can meet the requirement of high dynamic range.

[0077] In some embodiments of the present application, each pixel unit 8 in the pixel array corresponds to one light adjustment unit 9 in the light adjustment array, which means that a target pixel unit 8 in the pixel array corresponds to a target light adjustment unit 9 in the light adjustment array, that is, after the incident light is reflected by the target light adjustment unit 9 to the total reflector 2, the target pixel unit 8 in the image sensor 4 is reflected by the total reflector 2.

[0078] In some embodiments of the present application, the light adjustment array includes two or more light adjustment units 9, and different light adjustment units 9 can adjust light lines of different incident angles, respectively.

[0079] In some embodiments of the present application, the image sensor 4 includes two or more pixel units 8, and one pixel unit 8 corresponds to one light adjustment unit 9. Different light adjustment units 9 can individually adjust the brightness value of the pixel unit 8 corresponding thereto, so that the light adjustment assembly 3 can accurately adjust the brightness value of a single pixel point in the image collected by the image sensor 4, avoid overexposure caused by excessive light intensity in part of the image, and thus achieve the requirement of high dynamic range imaging and improve the imaging quality of the camera module 100.

[0080] In the embodiment of the present application, since the image sensor 4 can collect image data of the environment where the camera module 100 is located, the feedback circuit 5 can obtain a feedback signal based on the image data collected by the first light sensing area in the image sensor 4 and transmit it to the light adjustment assembly 3; the light adjustment assembly 3 can adjust the reflectivity of the first light adjustment area corresponding to the first light sensing area in the light adjustment assembly 3 based on the feedback signal. Therefore, the feedback circuit 5 can obtain a feedback signal based on the brightness value of the first light sensing area and transmit it to the light adjustment assembly 3, and the light adjustment assembly 3 can reduce or enhance the reflectivity of the first light adjustment area based on the feedback signal, thereby changing the intensity of the light reflected via the first light adjustment area, so that the light reflected to the total reflector 2 through the first light adjustment area of the light adjustment assembly 3, and then reflected to the first light sensing area of the image sensor 4 through the total reflector 2, the image sensor 4 can obtain an image with normal exposure, thereby avoiding overexposure of the first light sensing area due to excessive light intensity, and thus achieving the requirement of high dynamic range imaging and improving the imaging quality of the camera module 100. Moreover, when the camera module 100 of the present application is used, since multiple exposure HDR technology is not required for imaging, the camera module 100 does not need to fuse multiple images into one image when shooting images, thereby avoiding the problem that the synthesized image has motion blur due to the time difference between multiple images, and thus improving the imaging quality of the camera module 100. Moreover, the circuit structure of the image sensor 4 of the present application is simple and the manufacturing cost is low. At the same time, the camera module 100 of the present application can be used with different models of image sensors 4, and can achieve high dynamic range function on main camera, long focal, wide angle, front camera and other modules, and can achieve HDR shooting and recording functions in all scenes, which is conducive to volume multiplexing and thus reduces the cost.

[0081] In some embodiments of the present application, when the first light sensing area is a high-light area, the brightness value of the first light sensing area is larger, and the feedback circuit 5 can generate a feedback signal based on the brightness value of the first light sensing area and transmit it to the light adjustment assembly 3, and the light adjustment assembly 3 can reduce the reflectivity of the first light adjustment area based on the feedback signal, thereby changing the intensity of the light reflected via the first light adjustment area, so that the incident light is reflected to the total reflector 2 through the first light adjustment area of the light adjustment assembly 3, and then reflected to the first light sensing area of the image sensor 4 through the total reflector 2, and the brightness value of the first light sensing area is reduced correspondingly, so that the image sensor 4 can obtain an image with normal exposure, avoiding overexposure of the first light sensing area due to excessive light intensity when the first light sensing area is a high-light area.

[0082] In a possible example, as shown in FIG. 3, the dimming unit 9 includes a first polarizer 91, a second polarizer 92, and a liquid crystal module 93, the liquid crystal module 93 including a microlens 931, an electrode layer 932, a liquid crystal layer 933, a reflective layer 934, a circuit layer 935, and a substrate 936; the feedback signal includes a first voltage signal; and the feedback circuit 5 is specifically configured to apply the first voltage signal between the electrode layer 932 and the circuit layer 935 of the first dimming unit 9, so as to adjust the reflectivity of the first dimming unit 9, the first dimming unit 9 being a dimming unit 9 in the second region; and the incident light entering the dimming assembly 3 is transmitted to the reflective layer 934 in sequence through the first polarizer 91, the microlens 931, and the liquid crystal layer 933, and is emitted in sequence through the liquid crystal layer 933, the microlens 931, and the second polarizer 92 after being reflected by the reflective layer 934.

[0083] It can be understood that, by applying the first voltage signal between the electrode layer 932 and the circuit layer 935 of the first dimming unit 9, the feedback circuit 5 can control the voltage between the electrode layer 932 and the circuit layer 935, so as to control the deflection of the liquid crystal molecules in the liquid crystal layer 933, and further adjust the reflectivity of the first dimming unit 9. The polarization state of the light can be modulated through the first polarizer 91, the liquid crystal layer 933, and the second polarizer 92, and further the amplitude of the light can be modulated, so that when the incident light passes through the dimming assembly 3, the dimming assembly 3 can adjust the intensity of the incident light by modulating the amplitude of the light, to avoid the intensity of the incident light being too large to cause overexposure, thereby enabling the camera module 100 to meet the requirement of high dynamic range imaging. Meanwhile, the microlens 931 has good light condensing capability, so that the microlens 931 can converge more light, further reducing the loss of light, and improving the imaging quality of the camera module 100.

[0084] In some embodiments of the present application, the substrate 936 can be a printed circuit board (PCB).

[0085] In some embodiments of the present application, the electrode layer 932 can be a transparent electrode. In this way, it can be ensured that the light can pass through the electrode layer 932 for transmission, to avoid the electrode layer 932 hindering the transmission of the light.

[0086] In some embodiments of the present application, the circuit layer 935 can be a silicon-based circuit layer.

[0087] In another possible example, as shown in FIG. 4, the light adjustment unit 9 includes a first polarizer 91, a second polarizer 92, and a liquid crystal module 93, the liquid crystal module 93 including a glass cover plate 937, an electrode layer 932, a liquid crystal layer 933, a reflective layer 934, a circuit layer 935, and a substrate 936; the feedback signal includes a first voltage signal; and the feedback circuit 5 is specifically configured to apply the first voltage signal between the electrode layer 932 and the circuit layer 935 of the first light adjustment unit 9, so as to adjust the reflectivity of the first light adjustment unit 9, the first light adjustment unit 9 being the light adjustment unit 9 in the second region; and the incident light entering the light adjustment assembly 3 is transmitted to the reflective layer 934 in sequence through the first polarizer 91, the glass cover plate 937, and the liquid crystal layer 933, and is emitted in sequence through the liquid crystal layer 933, the glass cover plate 937, and the second polarizer 92 after being reflected by the reflective layer 934.

[0088] In a possible example, the electrode layer 932 is arranged at an edge position of the liquid crystal layer 933.

[0089] It can be understood that, when the electrode layer 932 is arranged at the edge position of the liquid crystal layer 933, the incident light can not pass through the electrode layer 932 when passing through the microlens 931 and entering the liquid crystal layer 933, so that the light energy loss caused by the light passing through the transparent electrode can be avoided, and the loss of the light passing through the light adjustment assembly 3 can be further reduced, and the imaging quality of the camera module 100 can be improved. Meanwhile, the driving voltage for driving the movement of the liquid crystal molecules in the liquid crystal layer 933 can be formed between the electrode layer 932 and the circuit layer 935, so that the reflectivity of the light adjustment assembly 3 can be adjusted by adjusting the voltage value between the electrode layer 932 and the circuit layer 935.

[0090] In some embodiments of the present application, the electrode layer 932 can be a ring electrode, and a ring-shaped groove can be formed at the edge position of the liquid crystal layer 933, and the ring electrode is nested in the ring-shaped groove.

[0091] In another possible example, the electrode layer 932 covers the liquid crystal layer 933.

[0092] It can be understood that, when the electrode layer 932 covers the liquid crystal layer 933, the driving voltage for driving the movement of the liquid crystal molecules in the liquid crystal layer 933 can be formed between the electrode layer 932 and the circuit layer 935, so that the reflectivity of the light adjustment assembly 3 can be adjusted by adjusting the voltage value between the electrode layer 932 and the circuit layer 935.

[0093] In some embodiments of the present application, the reflective layer 934 is a high-reflection coating layer.

[0094] It can be understood that the high reflection coating is a coating structure with high reflectivity, when the reflection layer 934 is a high reflection coating, the reflectivity of the reflection layer 934 can be improved, and the loss of light energy in the reflection process can be avoided, thereby further reducing the loss of light when passing through the light adjusting assembly 3, and improving the imaging quality of the camera module 100.

[0095] In some embodiments of the present application, the light adjusting assembly 3 is one of a liquid crystal on silicon (LCoS) chip, a spatial light modulator, and a micro-mirror array.

[0096] In some embodiments of the present application, the LCoS chip is a silicon-based liquid crystal chip, which has a liquid crystal layer 933. The amplitude of the light can be controlled through the liquid crystal layer 933 of the chip, thereby adjusting the intensity of the light. The light adjusting assembly 3 can adjust the intensity of the incident light, thereby avoiding excessive exposure caused by excessive intensity of the incident light, so that the camera module 100 can meet the requirements of high dynamic range imaging.

[0097] In some embodiments of the present application, the spatial light modulator is a device that modulates the spatial distribution of light waves. Under the control of an electrical driving signal, the amplitude or intensity, phase, polarization state, etc. of the spatial light distribution are changed. The spatial light modulator is a key device in the fields of real-time optical information processing, adaptive optics, and optical computing of modern optics. In this way, the amplitude of the light can be controlled through the spatial light modulator, thereby adjusting the intensity of the light. The light adjusting assembly 3 can adjust the intensity of the incident light, thereby avoiding excessive exposure caused by excessive intensity of the incident light, so that the camera module 100 can meet the requirements of high dynamic range imaging.

[0098] In some embodiments of the present application, the micro-mirror array is also called a digital micromirror device (DMD). The micro-mirror array includes a plurality of mirror units arranged in an array. Each mirror unit includes a mirror and a mechanical structure for driving the mirror to deflect. The deflection angle of the mirror can be controlled by the mechanical structure, the transmission direction of the light reflected by the mirror can be controlled, and the number of light beams reflected to the total reflector 2 can be controlled, thereby adjusting the reflectivity of the mirror unit. The light adjusting assembly 3 can adjust the intensity of the incident light, thereby avoiding excessive exposure caused by excessive intensity of the incident light, so that the camera module 100 can meet the requirements of high dynamic range imaging.

[0099] In some embodiments of the present application, as shown in FIG. 1, the camera module 100 further includes a second lens assembly 7, which is arranged between the total reflector 2 and the image sensor 4. The incident light reflected by the total reflector 2 is converged and transmitted to the image sensor 4 through the second lens assembly 7.

[0100] It can be understood that, by converging the light rays through the second lens assembly 7, the incident light rays reflected by the total reflection mirror 2 can be converged on the image sensor 4 for imaging when passing through the second lens assembly 7, thereby improving the imaging quality of the image sensor 4. Meanwhile, the second lens assembly 7 can compensate for the aberration of the first lens group, thereby improving the resolving power of the imaging system. For example, when an aberration (such as coma, astigmatism, etc.) of the first lens assembly 1 is +A, the second lens assembly 7 can be changed in curvature, thickness, lens spacing, etc. to generate an aberration of -A, so that the aberration can be compensated by superimposing the first lens assembly 1 and the second lens assembly 7. In addition, the second lens assembly 7 can also achieve chief ray angle (CRA) matching. For example, by changing the curvature distribution of the second lens group, the incident angle of the chief ray can be controlled to meet the chief ray angle matching requirements of the image sensor 4, thereby avoiding the crosstalk between the pixel units 8 of the image sensor 4 and the color cast phenomenon caused by the mismatch of the chief ray angle.

[0101] In some embodiments of the present application, the camera module 100 further includes an infrared cut-off filter 6, which is arranged between the second lens assembly 7 and the image sensor 4. The incident light rays transmitted through the second lens assembly 7 enter the image sensor 4 after being filtered by the infrared cut-off filter 6. In this way, the infrared cut-off filter 6 can filter out the infrared band invisible to the human eye, thereby improving the imaging effect of the camera module 100.

[0102] In some embodiments of the present application, as shown in FIG. 5, the feedback circuit 5 includes a calculation module 51, which is configured to calculate the feedback value of the feedback signal based on the image data collected by the first light sensing area of the image sensor 4.

[0103] In some embodiments of the present application, as shown in FIG. 6, the calculation module 51 can include a clock circuit, a counter, a feedback signal mapping unit, a matrix multiplication calculation unit, and a memory (EEPROM). The calculation process of the calculation module 51 can be as follows: first, the electrical signal of the first light sensing area is read by the analog-to-digital converter of the image sensor 4, and the electrical signal is transmitted to the feedback signal mapping unit. The feedback signal mapping unit calculates the feedback value of the feedback signal according to the mapping relationship. Second, during the reading of the first light sensing area by the column analog-to-digital converter of the image sensor 4, the clock circuit obtains a clock signal, the counter obtains the position information of the first light sensing area according to the clock signal, and the position information is transmitted to the matrix multiplication calculation unit. The matrix multiplication calculation unit can obtain the position information (i.e., coordinate information) of the first light adjustment area according to the preset solving matrix. The feedback value of the feedback signal and the position information of the first light adjustment area can be stored in the memory.

[0104] In some embodiments of the present application, as shown in FIG. 5, the feedback circuit 5 can further include an encoding module 52 connected with the calculation module 51. The encoding module 52 can encode the feedback value of the feedback signal calculated by the calculation module 51 and the position information of the first dimming region into several bytes to obtain encoded data.

[0105] In some embodiments of the present application, as shown in FIG. 5, the feedback circuit 5 can further include a decoding module 53. The decoding module 53 can decode the encoded data generated by the encoding module 52 to obtain decoded data. The decoded data includes the feedback value of the feedback signal calculated by the calculation module 51 and the position information of the first dimming region.

[0106] In some embodiments of the present application, the feedback circuit 5 can further include a transmission line for transmitting the encoded data to the decoding module 53. For example, the encoding module 52 and the decoding module 53 can transmit data through an 8-bit bus.

[0107] It can be understood that, by encoding and decoding the feedback value of the feedback signal calculated by the calculation module 51 and the position information of the first dimming region, the transmission speed of the feedback signal can be improved to realize fast real-time feedback.

[0108] In some embodiments of the present application, as shown in FIG. 5, the feedback circuit 5 can further include a driving module 54, i.e., a driving IC, connected with the decoding module 53. The driving module 54 can obtain the feedback signal based on the decoded data. The driving module 54 can include a row selector and a column selector. The driving module 54 can receive the decoded data generated by the decoding module 53, find the position of the first dimming region in the dimming assembly 3 through the row selector and the column selector according to the position information of the first dimming region in the decoded data, obtain the feedback signal according to the feedback value of the feedback signal in the decoded data, and transmit the feedback signal to the dimming assembly 3. When the dimming assembly 3 includes an electrode layer 932, a liquid crystal layer 933, and a circuit layer 935, the feedback signal is a voltage signal applied between the electrode layer 932 and the circuit layer 935.

[0109] In some embodiments of the present application, as shown in FIG. 6, the circuit of one photosensitive region 41 in the image sensor 4 can include a parasitic capacitor FD, a reset switch RST, an analog switch TG, a voltage follower SF, a row selection switch RSL, and a photodiode. The first end of the reset switch RST is connected to the first end of the voltage follower SF, the second end of the reset switch RST is connected to the first end of the analog switch TG, the second end of the analog switch TG is connected to the photodiode, the second end of the voltage follower SF is connected to the first end of the row selection switch RSL, and the second end of the RSL is grounded. The third end of the voltage follower SF is connected between the second end of the reset switch RST and the first end of the analog switch TG, the first end of the parasitic capacitor FD is connected between the third end of the voltage follower SF and the second end of the reset switch RST, and the second end of the parasitic capacitor FD is grounded. In this way, when light shines on the photodiode, the photodiode will generate electron-hole pairs, and the electron-hole pairs will separate under the action of the electric field of the photodiode, that is, the electrons move to the n region of the diode, and the holes move to the p region of the diode, thereby generating a current and transmitting it to the analog switch TG. The analog switch TG can control the current to flow to the parasitic capacitor FD, so that the charge can be stored in the parasitic capacitor FD. The reset switch RST can control the parasitic capacitor FD to reset to a high level, facilitating the reading of the charge in the parasitic capacitor FD. The charge in the parasitic capacitor FD can be transmitted to the voltage follower SF after being read out, amplified by the voltage follower SF, and transmitted to the row selection switch RSL, and the electrical signal is output through the row selection switch RSL, so that the optical signal can be converted into an electrical signal.

[0110] The present application also provides an electronic device, which comprises the camera module 100 in the above embodiments.

[0111] The present application provides an electronic device, which comprises the camera module 100 in the above embodiments. Since the image sensor of the camera module 100 can collect image data of the environment where the camera module is located, the feedback circuit can generate a feedback signal based on the image data collected by the first photosensitive region of the image sensor and transmit it to the light adjustment assembly; the light adjustment assembly can adjust the reflectivity of the first light adjustment region corresponding to the first photosensitive region in the light adjustment assembly based on the feedback signal, so that the electronic device can meet the requirements of high dynamic range imaging and improve the imaging quality of the electronic device.

[0112] The image processing method provided in the embodiments of the present application can be executed by a camera module, or an image processing device, or an electronic device, or a functional module or entity in the electronic device. The electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, or the like, and the electronic device includes a camera module. In the embodiments of the present application, the image processing method is executed by the image processing device as an example to illustrate the image processing method provided in the embodiments of the present application.

[0113] FIG. 7 shows a schematic diagram of an image processing method provided in the embodiments of the present application. As shown in FIG. 7, the image processing method provided in the embodiments of the present application can include the following steps 101 to 104.

[0114] In step 101, the image processing device controls a feedback circuit of the camera module to acquire first image data collected by an image sensor in the camera module in a first photosensitive region.

[0115] In some embodiments of the present application, when light enters the image sensor and irradiates the first photosensitive region, the first photosensitive region can convert the light signal into an electrical signal. The electrical signal can be amplified through a signal amplification circuit, and the amplified electrical signal can be converted into a digital signal through an analog-to-digital conversion circuit. The digital signal can be transmitted to a processing module of the image sensor through a Mobile Industry Processor Interface (MIPI).

[0116] In some embodiments of the present application, the image data collected by the image sensor in the first photosensitive region can include a brightness value of the first photosensitive region and first position information. The first position information includes a row coordinate and a column coordinate of the first photosensitive region. It should be noted that the row coordinate and the column coordinate of the first photosensitive region refer to the coordinates of a single pixel unit in a pixel array of the image sensor.

[0117] In step 102, the image processing device controls a feedback circuit in the camera module to obtain a first feedback signal based on the first image data.

[0118] In some embodiments of the present application, for the structure of the light adjusting unit of the light adjusting assembly shown in FIGS. 3 and 4, the image processing device can generate second position information of a first light adjusting region based on the first position information of the first photosensitive region, and then generate a feedback value corresponding to the second position information based on the brightness value of the first photosensitive region. That is, the feedback signal transmitted by the image processing device to the light adjusting assembly carries the second position information of the first light adjusting region and the feedback value corresponding to the second position information.

[0119] It can be understood that the image processing apparatus generates the feedback signal based on the image data of the first photosensitive region, and transmits the feedback signal to the dimming component, so that the dimming component can adjust the reflectivity of the first dimming region based on the feedback signal of the first photosensitive region, thereby the dimming component can adjust the brightness value of the first photosensitive region to avoid overexposure of the first photosensitive region, and further the image processing apparatus can meet the requirement of high dynamic range imaging and improve the imaging quality of the image processing apparatus.

[0120] In some embodiments, as shown in FIG. 8, the image processing apparatus in the above-mentioned step 102 generates the feedback signal based on the image data, which can be implemented by the following steps 102a to 102c.

[0121] In step 102a, the image processing apparatus determines the average brightness value of the first photosensitive region based on the first image data.

[0122] In some embodiments of the present application, when the incident light enters the image sensor and irradiates the first photosensitive region, the first photosensitive region can convert the optical signal into an electrical signal, the electrical signal can be amplified by the signal amplification circuit, and the amplified electrical signal can be converted into a digital signal by the analog-to-digital conversion circuit. At this time, the image processing apparatus can obtain the average brightness value by calculating the average value of the digital signal.

[0123] In step 102b, the image processing apparatus generates a first digital signal based on the average brightness value and a first functional relationship.

[0124] In some embodiments of the present application, the above-mentioned first functional relationship can be a linear function relationship, and the above-mentioned first functional relationship is used to represent the mapping relationship between the signal mean value of the photosensitive region of the image sensor and the feedback value of the feedback signal. That is, the above-mentioned first functional relationship is used to represent the mapping relationship between the brightness value of the photosensitive region of the image sensor and the voltage value required to be applied to the corresponding dimming region.

[0125] In some embodiments of the present application, FIG. 9 shows a schematic diagram of the mapping relationship between the signal mean value of the photosensitive region of the image sensor and the feedback value of the feedback signal.

[0126] In step 102c, the image processing apparatus obtains a first feedback signal based on the first digital signal.

[0127] It can be understood that the image processing apparatus determines the first voltage value and generates the first digital signal based on the average brightness value and the first functional relationship, and the image processing apparatus obtains the first feedback signal based on the first digital signal, so that the image processing apparatus can obtain the first feedback signal based on the first image data.

[0128] In step 103, the image processing apparatus controls the light adjustment component in the camera module to adjust the reflectivity of a first light adjustment region corresponding to the first light sensing region in the light adjustment component based on the first feedback signal.

[0129] In step 104, the image processing apparatus controls the image sensor to capture the second image data.

[0130] It can be understood that the image processing apparatus determines the first voltage value and generates the first digital signal based on the average brightness value and the first function relationship, the image processing apparatus obtains the first feedback signal based on the first digital signal, and the image processing apparatus controls the light adjustment component in the camera module to adjust the reflectivity of the first light adjustment region corresponding to the first light sensing region in the light adjustment component based on the first feedback signal, so that the light adjustment component can adjust the intensity of the incident light reflected via the first light adjustment region, and the image sensor can obtain the second image data with normal exposure after the incident light is reflected to the total reflector via the first light adjustment region of the light adjustment component, and then reflected to the first light sensing region of the image sensor from the total reflector, which can avoid overexposure of the first light sensing region due to excessively high light intensity, achieve the requirement of high dynamic range imaging, and improve the imaging quality of the camera module.

[0131] In some embodiments, as shown in FIG. 10, the image processing method provided by the embodiments of the present application can be implemented by the following steps S101 to S104, taking the image processing apparatus as an example.

[0132] In step S101, the camera module calibrates the first position information of the first light sensing region and the second position information of the first light adjustment region.

[0133] In some embodiments of the present application, the first position information of the first light sensing region and the second position information of the first light adjustment region are calibrated, specifically including: establishing a spatial mapping relationship between the first position information (x, y) of the first light sensing region and the second position information (u, v) of the first light adjustment region, and the spatial mapping relationship can be calculated based on the following formula 1:

[0134] wherein x is the row coordinate of the first light sensing region, y is the column coordinate of the first light sensing region, u is the row coordinate of the first light adjustment region, v is the column coordinate of the first light adjustment region, T is a solution matrix, t11 is the value of the first row and the first column of the solution matrix T, t12 is the value of the first row and the second column of the solution matrix T, t13 is the value of the first row and the third column of the solution matrix T, t21 is the value of the second row and the first column of the solution matrix T, t22 is the value of the second row and the second column of the solution matrix T, t23 is the value of the second row and the third column of the solution matrix T, t31 is the value of the third row and the first column of the solution matrix T, t32 is the value of the third row and the second column of the solution matrix T, and t33 is the value of the third row and the third column of the solution matrix T.

[0135] As can be seen from formula 1, when the first position information (x, y) of the first light sensing area and the solving matrix T are known, the second position information (u, v) of the first light modulation area corresponding to the first light sensing area can be obtained, so that the feedback circuit transmits the feedback signal to the first light modulation area according to the second position information. Therefore, the key of the above module calibration lies in calculating the value of the solving matrix T.

[0136] In some embodiments of the present application, as shown in FIG. 11, the method of the above module calibration can be a checkerboard projection method, by which the value of the solving matrix T can be calculated.

[0137] Exemplarily, the checkerboard method is a method of obtaining the internal parameters of a camera by using Zhang Zhengyou calibration method, establishing constraint equation sets by using the consistency of a checkerboard plane in two coordinate systems, solving the initial solution of the external parameters of the two coordinate systems by using a linear method, and further optimizing by using a nonlinear optimization method.

[0138] Generally, the specific process of calibration by using the checkerboard projection method can be as follows: first, drive the light modulation assembly to display a specific checkerboard pattern, and read out the image after imaging between the second lens group through the image sensor. Calculate the corner point coordinates by using mature detection algorithm, and then obtain the mapping relationship between the corner points, and solve the matrix T based on formula 1. At the same time, burn the value of the solved matrix T into the memory (EEPROM) for use by the calculation module of the feedback circuit.

[0139] S102, the image sensor senses light and processes data.

[0140] In some embodiments of the present application, the image sensor converts the optical signal into an electrical signal after sensing light, the electrical signal can be amplified by a signal amplification circuit, the amplified electrical signal can be converted into a digital signal by an analog-to-digital conversion circuit, and the digital signal can be transmitted to the processing module of the image sensor through a mobile industry processor interface (Mobile Industry Processor Interface, MIPI), and the processing module processes to obtain image data.

[0141] S103, the feedback circuit of the camera module generates a feedback signal based on the image data and transmits it to the light modulation assembly.

[0142] In some embodiments of the present application, the feedback circuit of the camera module includes a calculation module. The processing module can transmit the readout data of the current frame of the image to the calculation module of the feedback circuit. The calculation module can perform block processing on the RAW format image, and calculate the position coordinates (x, y) and signal mean SIG MEAN of each pixel block (block), i.e., the first position information (x, y) and the average brightness value of the first light sensing area. Further, the calculation module can calculate the second position information (u, v) of the first light adjustment area corresponding to the first light sensing area according to the above formula 1, and at the same time, the calculation module can calculate the size of the feedback signal SIG FEEDBACK according to the signal mean SIG MEAN and the first function relationship. The first function relationship is used to represent the mapping relationship between the brightness value of the different light sensing areas of the image sensor and the voltage value required to be applied to the corresponding light adjustment area.

[0143] In some embodiments of the present application, as shown in FIG. 12, the feedback circuit of the camera module further includes an encoding module and a decoding module. The encoding module can encode and package the second position information (u, v) and the feedback signal SIG FEEDBACK into 2-byte data, and transmit the 2-byte data to the decoding module through the data bus. The 2-byte data includes a start identifier, the second position information (u, v), the feedback signal SIG FEEDBACK and an end identifier distributed in sequence.

[0144] In some embodiments of the present application, the feedback circuit of the camera module further includes a driving module. The driving module transmits voltage to the first light adjustment area of the light adjustment assembly according to the decoding data of the decoding module. For the structure of the light adjustment unit of the light adjustment assembly shown in FIGS. 3 and 4, the driving module can apply voltage between the electrode layer and the circuit layer according to the decoding data, so as to adjust the reflectivity of the first light adjustment area by adjusting the inter-electrode voltage between the electrode layer and the circuit layer. That is, when the signal value of the feedback signal is small, the driving module applies a larger voltage between the electrode layer and the circuit layer to increase the reflectivity of the first light adjustment area; when the signal value of the feedback signal is large, the driving module applies a smaller voltage between the electrode layer and the circuit layer to reduce the reflectivity of the first light adjustment area, thereby realizing adaptive control of the exposure of the image sensor and improving the dynamic range of the camera module.

[0145] S104, after the light adjustment assembly adjusts the reflectivity, the light enters the image sensor again and generates a target image. The target image is processed by an image signal processing (ISP) module of the image, and the target image is compressed.

[0146] In some embodiments of the present application, since there is a significant difference between the photosensitive characteristics of the image sensor and the human eye, the target image in RAW format needs to be further processed to meet the preferences of the human eye. After the image sensor generates the target image, the target image can be sent to the ISP module for processing, and after noise reduction, white balance, color calibration, gamma transformation and other operations, the image in JPEG format is compressed for storage or transmission to the display. Gamma transformation is an image enhancement method that adjusts the contrast of the image and enhances the details of the dark or bright parts.

[0147] In some embodiments of the present application, as shown in FIG. 13, the specific process of the above image processing method can be:

[0148] Firstly, the position mapping relationship between the first photosensitive area and the first light adjustment area is determined by the above module calibration method, and the position mapping relationship is stored in the storage (EEPROM).

[0149] Secondly, the image sensor obtains the first position information and the average brightness value of the first photosensitive area, the calculation module calculates the second position information of the first light adjustment area based on the first position information and the position mapping relationship, and calculates the feedback value of the feedback signal corresponding to the second position information based on the average brightness value and the first function relationship. Then, the encoding module is encoded, the decoding module is decoded, so that the feedback signal is transmitted to the driving module, and the driving module adjusts the voltage value between the electrode layer and the circuit layer of the light adjustment assembly according to the second position information and the feedback value of the feedback signal corresponding to the second position information, so as to adjust the reflectivity of the light adjustment assembly, and then adjust the intensity of the light passing through the light adjustment assembly. The light adjusted is reflected to the first photosensitive area of the image sensor again through the total reflection mirror, so that the first photosensitive area can collect image data with normal exposure, and then the image sensor can collect the target image with normal exposure.

[0150] Finally, the target image is transmitted to the ISP module through the Mobile Industry Processor Interface (MIPI), the target image is processed by the ISP module, and the target image is compressed to obtain an image that meets the preferences of the human eye.

[0151] The image processing method provided in the embodiments of the present application can be executed by the image processing device. In the embodiments of the present application, the image processing device executes the image processing method as an example to illustrate the image processing device provided in the embodiments of the present application.

[0152] FIG. 14 shows a possible structure of a photographing device involved in the embodiments of the present application. As shown in FIG. 14, the image processing device 300 can include the camera module 100 in the above embodiments, and the image processing device 300 can further include:

[0153] The processing module 301 is configured to control a feedback circuit of the camera module to obtain first image data collected by a first photosensitive region of an image sensor in the camera module.

[0154] The processing module 301 is further configured to control the feedback circuit in the camera module to obtain a first feedback signal based on the first image data.

[0155] The processing module 301 is further configured to control a dimming component in the camera module to adjust reflectivity of a first dimming region corresponding to the first photosensitive region in the dimming component based on the first feedback signal.

[0156] The processing module 301 is further configured to control the image sensor to collect second image data.

[0157] The embodiment of the present application provides an image processing device 300, which comprises the camera module 100 in the above embodiment. Since the image sensor of the camera module 100 can collect image data of an environment where the camera module is located, the feedback circuit can generate a feedback signal based on the image data collected by the first photosensitive region of the image sensor and transmit the feedback signal to the dimming component; and the dimming component can adjust the reflectivity of the first dimming region corresponding to the first photosensitive region in the dimming component based on the feedback signal, so that the electronic device can meet the requirement of high dynamic range imaging and improve the imaging quality of the electronic device.

[0158] In some embodiments of the present application, the processing module 301 is specifically configured to: determine an average brightness value of the first photosensitive region based on the first image data; generate a first digital signal based on the average brightness value and a first functional relationship; and obtain the first feedback signal based on the first digital signal.

[0159] The image processing apparatus 300 in the embodiments of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices than the terminal. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and the like, and can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like, and the embodiments of the present application are not limited thereto.

[0160] The image processing apparatus 300 in the embodiments of the present application can be an apparatus with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, and the embodiments of the present application are not limited thereto.

[0161] The photographing apparatus provided in the embodiments of the present application can implement each process implemented by the method embodiments of FIGS. 1 to 13, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0162] In some embodiments, as shown in FIG. 15, the embodiments of the present application further provide an electronic device 400, which includes a processor 401 and a memory 402, and the memory 402 has a program or instructions stored thereon, which can be run on the processor 401. When the program or instructions are executed by the processor 401, each process step of the photographing method embodiments described above is implemented, and the same technical effects are achieved. To avoid repetition, details are not described herein.

[0163] It should be noted that the electronic device in the embodiments of the present application includes the mobile electronic device and the non-mobile electronic device described above.

[0164] FIG. 16 is a schematic diagram of a hardware structure of an electronic device for implementing the embodiments of the present application.

[0165] The electronic device 500 includes, but is not limited to, a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, and a processor 510, and the like. In addition, the electronic device 500 further includes the camera module 100. It should be noted that the functions and connection relationships of the modules in the camera module 100 described above can refer to the description above, and will not be described here.

[0166] Those skilled in the art can understand that the electronic device 500 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 510 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management through the power management system. The electronic device structure shown in FIG. 13 does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than shown, or combine certain components, or different component arrangements, which will not be described here.

[0167] The processor 510 is configured to control the feedback circuit in the camera module 100 to acquire first image data collected by an image sensor in the camera module 100 in a first light sensing area; control the feedback circuit in the camera module 100 to obtain a first feedback signal based on the first image data; control the light adjustment component in the camera module 100 to adjust the reflectivity of a first light adjustment area corresponding to the first light sensing area in the light adjustment component based on the first feedback signal; and control the image sensor to collect second image data.

[0168] The electronic device 500 provided by the embodiments of the present application includes the camera module 100 in the above embodiments. Since the image sensor of the camera module 100 can collect image data of the environment where the camera module is located, the feedback circuit can generate a feedback signal based on the image data collected by the first light sensing area in the image sensor and transmit it to the light adjustment component; the light adjustment component can adjust the reflectivity of the first light adjustment area corresponding to the first light sensing area in the light adjustment component based on the feedback signal, so that the electronic device 500 can meet the requirements of high dynamic range imaging and improve the imaging quality of the electronic device 500.

[0169] In some embodiments of the present application, the processor 510 is specifically configured to determine an average brightness value of the first light sensing area based on the first image data; generate a first digital signal based on the average brightness value and a first function relationship; and obtain the first feedback signal based on the first digital signal.

[0170] In some embodiments of the present application, the processor 510 is specifically configured to determine an average brightness value of the first light sensing region based on the image data; determine a first voltage value based on the average brightness value and a first function relationship, and generate a feedback signal; the first function relationship is used to represent a mapping relationship between brightness values of different light sensing regions of the image sensor and voltage values required to be applied to corresponding dimming regions.

[0171] It should be understood that in the embodiments of the present application, the input unit 504 can include a graphics processing unit (GPU) 5041 and a microphone 5042. The graphics processing unit 5041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 506 can include a display panel 5061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 507 includes at least one of a touch panel 5071 and other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 can include two parts of a touch detection device and a touch controller. The other input devices 5072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), a trackball, a mouse, an operation lever, and the like, which will not be described here.

[0172] The memory 509 can be used to store software programs and various data. The memory 509 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 509 can include a volatile memory or a non-volatile memory, or the memory 509 can include both a volatile memory and a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch Link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 509 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0173] The processor 510 can include one or more processing units; in some embodiments, the processor 510 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 510.

[0174] The embodiments of the present application also provide a readable storage medium, the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize various processes of the above-mentioned focusing method embodiments and achieve the same technical effects. To avoid repetition, details are not described here.

[0175] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0176] The embodiment of the present application further provides a chip, which comprises a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running programs or instructions to realize the processes of the above focusing method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0177] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system level chip, a system chip, a chip system or a system on chip, etc.

[0178] The embodiment of the present application provides a computer program product, which is stored in a storage medium, and is executed by at least one processor to realize the processes of the above focusing method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0179] It should be noted that in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to the order of performing the functions as shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in the opposite order, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned example methods can be realized by means of software and a necessary general hardware platform, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a computer software product in essence or in the form of a part that contributes to the prior art, which is stored in a storage medium (such as a ROM / RAM, a magnetic disc, an optical disc), and includes a plurality of instructions for causing a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0181] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative and not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A camera module, comprising a first lens assembly, a total reflection mirror, a light modulation assembly, an image sensor and a feedback circuit, the first lens assembly, the total reflection mirror and the light modulation assembly are sequentially arranged, and the image sensor is arranged opposite to the total reflection mirror; incident light rays converge through the first lens assembly and then enter the light modulation assembly through the total reflection mirror, the light modulation assembly modulates the incident light rays and then reflects them to the total reflection mirror, and the total reflection mirror reflects the incident light rays to the image sensor; the image sensor comprises at least one photosensitive region, and the light modulation assembly comprises at least one light modulation region, one photosensitive region corresponds to one light modulation region; the feedback circuit is connected with the image sensor and the light modulation assembly respectively; the feedback circuit is used for obtaining a feedback signal based on image data collected by a first photosensitive region in the image sensor; the light modulation assembly is used for adjusting the reflectivity of a first light modulation region in the light modulation assembly corresponding to the first photosensitive region based on the feedback signal; the image sensor comprises a pixel array, and the pixel array comprises at least one pixel unit; the light modulation assembly comprises a light modulation array, and the light modulation array comprises at least one light modulation unit; each pixel unit in the pixel array corresponds to one light modulation unit in the light modulation array; the light modulation unit comprises a first polarizer, a second polarizer and a liquid crystal module, and the liquid crystal module comprises a microlens, an electrode layer, a liquid crystal layer, a reflecting layer, a circuit layer and a substrate; the feedback signal comprises a first voltage signal; the feedback circuit is specifically used for applying the first voltage signal between the electrode layer and the circuit layer of a first light modulation unit to adjust the reflectivity of the first light modulation unit, the first light modulation unit being a light modulation unit in the first light modulation region; incident light rays entering the light modulation assembly are transmitted to the reflecting layer through the first polarizer, the microlens and the liquid crystal layer in sequence, and then are emitted through the second polarizer after being reflected by the reflecting layer; the electrode layer is arranged at an edge position of the liquid crystal layer, or the electrode layer covers the liquid crystal layer; the light modulation assembly is one of a liquid crystal on silicon (LCoS) chip, a spatial light modulator and a micromirror array; the total reflection mirror comprises oppositely arranged first and second mirror bodies, the first mirror body is arranged opposite to the first lens assembly, and the second mirror body has a reflecting surface; incident light rays through the first lens assembly are transmitted to the light modulation assembly through the first and second mirror bodies in sequence, are reflected to the reflecting surface of the second mirror body after being modulated by the light modulation assembly, and then are reflected to the image sensor through the reflecting surface; the camera module further comprises a driving device, and the driving device is used for driving the light modulation assembly to move so as to adjust an included angle between an exit surface of the light modulation assembly and an optical axis of the first lens assembly; and the camera module further comprises a second lens assembly, and the second lens assembly is arranged between the total reflection mirror and the image sensor. ​ ​ ​ ​ 2. The camera module of claim 1, wherein, ​ ​ ​ 3. The camera module of claim 2, wherein, ​ ​ ​ ​ 4. The camera module of claim 3, wherein, ​ 5. The camera module of claim 1 or 2, wherein, ​ 6. The camera module of claim 1, wherein, ​ ​ 7. The camera module of claim 6, wherein, ​ 8. The camera module of claim 1, wherein, ​ The incident light reflected by the total reflection mirror is converged and transmitted to the image sensor by the second lens assembly.

9. The camera module of claim 1, wherein, The feedback circuit comprises a calculation module, an encoding module, a transmission line, a decoding module and a driving module connected in sequence. The calculation module is configured to calculate a feedback value of the feedback signal based on image data collected by a first photosensitive region of the image sensor. The encoding module is configured to encode the feedback value to obtain encoded data. The transmission line is configured to transmit the encoded data to the decoding module. The decoding module is configured to decode the encoded data to obtain decoded data. The driving module is configured to obtain a feedback signal based on the decoded data. 10.An electronic device comprising the camera module according to any one of claims 1 to 9. 11.An image processing method performed by an electronic device comprising the camera module according to any one of claims 1 to 9. The method comprises: controlling a feedback circuit of the camera module to obtain first image data collected by a first photosensitive region of an image sensor in the camera module; controlling the feedback circuit in the camera module to obtain a first feedback signal based on the first image data; controlling a light adjustment assembly in the camera module to adjust reflectivity of a first light adjustment region corresponding to the first photosensitive region in the light adjustment assembly based on the first feedback signal; controlling the image sensor to collect second image data.

12. The image processing method of claim 11, wherein, The controlling the feedback circuit in the camera module to obtain the first feedback signal based on the first image data comprises: determining an average brightness value of the first photosensitive region based on the first image data; generating a first digital signal based on the average brightness value and a first functional relationship; obtaining the first feedback signal based on the first digital signal. 13.An image processing apparatus comprising the camera module according to any one of claims 1 to 9. The image processing apparatus further comprises: a processing module configured to control a feedback circuit of the camera module to obtain first image data collected by a first photosensitive region of an image sensor in the camera module; the processing module is further configured to control the feedback circuit in the camera module to obtain a first feedback signal based on the first image data; the processing module is further configured to control a light adjustment assembly in the camera module to adjust reflectivity of a first light adjustment region corresponding to the first photosensitive region in the light adjustment assembly based on the first feedback signal; the processing module is further configured to control the image sensor to collect second image data.

14. The image processing apparatus according to claim 13, wherein The processing module is specifically configured to: determine an average brightness value of the first photosensitive region based on the first image data; generate a first digital signal based on the average brightness value and a first functional relationship; obtain the first feedback signal based on the first digital signal. 15.An electronic device comprising a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the image processing method according to claim 11 or 12.

16. A computer readable storage medium, on which a computer program is stored, which computer program, when executed by a processor, implements the steps of the information processing method according to claim 11 or 12.

17. A chip, comprising a processor and a communication interface, the communication interface and the processor being coupled, the processor being configured to run a program or an instruction, and implement the method according to claim 11 or 12.

Images