Image sensor, image processing system including the same, and method of operating image processing system
The image processing system in smartphones automatically adjusts settings for auto exposure and focus based on front and rear camera information to enhance image capture on the user's region of interest, addressing the limitations of existing systems by detecting gaze direction and reducing power consumption.
Patent Information
- Application Number
- JP2024014351
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-10-26
- Filing Date
- 2024-02-01
- Publication Date
- 2024-11-08
AI Technical Summary
Existing image processing systems in smartphones lack the ability to automatically adjust settings for auto exposure and auto focus based on information from both rear and front cameras, and fail to detect a user's region of interest for enhanced image capture.
An image processing system that includes a first image sensor for rear camera and a second image sensor for front camera, where the second sensor detects the user's gaze direction and generates probability maps to adjust settings for the first sensor to focus on the user's region of interest, reducing power consumption by operating in low power mode when not in use.
Automatically detects and adjusts settings for auto white balance, exposure, and focus on the user's desired region of interest, enhancing image capture without manual operation and reducing power consumption.
Smart Images

Figure 2024159484000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to semiconductor design techniques, and more particularly to an image sensor, an image processing system including the same, and a method of operating the image processing system. [Background technology]
[0002] An image sensor is a device that captures an image using the properties of a semiconductor that reacts to light. The image sensors can be roughly divided into image sensors using a charge coupled device (CCD) and image sensors using a complementary metal oxide semiconductor (CMOS). In recent years, image sensors using CMOS have been widely used due to the advantage that analog and digital control circuits can be directly implemented on one integrated circuit (IC).
[0003] Recently, image processing systems such as smartphones include multiple image sensors. The smartphone includes, for example, a first image sensor corresponding to a rear camera and a second image sensor corresponding to a front camera. The first and second image sensors operate independently or dependently on each other depending on their functions. Summary of the Invention [Problem to be solved by the invention]
[0004] An embodiment of the present invention provides an image processing system and an operating method thereof that can automatically adjust setting information required for functions (e.g., auto white balance, auto exposure, and auto focus) of a first image sensor corresponding to a rear camera based on an information image generated from a second image sensor corresponding to a front camera.
[0005] Furthermore, the present invention provides an image sensor that can automatically detect a region of interest desired by a user from a captured image. [Means for solving the problem]
[0006] According to one aspect of the present invention, an image sensor can include a first detection circuit for detecting an eye area of a user from an information image of the user, a first probability map generation circuit for generating a first probability map corresponding to the direction in which the user's eyes are gazed based on the eye area, and a storage circuit for storing the first probability map.
[0007] According to another aspect of the present invention, an image processing system may include a first image sensor for performing at least one function on a region of interest that a user gazes at based on setting information and acquiring a captured image including the region of interest, a second image sensor for generating a first probability map corresponding to the direction in which the user's eyes are gazed based on an information image acquired by photographing the user, and an image processor for detecting the region of interest based on the captured image and the first probability map and adjusting the setting information based on the region of interest.
[0008] According to yet another aspect of the present invention, a method of operating an image processing system may include the steps of: when a rear camera is enabled and enters a normal mode, a front camera is enabled and enters a low power mode; the front camera photographs a user and generates a first probability map of a direction in which the user's eyes are fixed; an image processor detects a region of interest in which the user's eyes are fixed based on the first probability map and adjusts setting information corresponding to the region of interest; and the rear camera performs at least one function on the region of interest based on the setting information to obtain a captured image including the region of interest. Effect of the Invention
[0009] An embodiment of the present invention has the advantage that a user can easily obtain a desired image without manual operation by automatically detecting a region of interest desired by the user and automatically performing scheduled operations (e.g., white balancing, exposure, focusing, etc.) on the region of interest. [Brief description of the drawings]
[0010] [Figure 1] 1 is a block diagram of an image processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a block diagram of a first image sensor shown in FIG. 1. [Diagram 3] 2 is a block diagram of a second image sensor shown in FIG. 1; [Figure 4] FIG. 4 is a block diagram of a second digital block shown in FIG. 3. [Diagram 5] FIG. 2 is a block diagram of the image processor shown in FIG. 1. [Figure 6] 4 is a flowchart illustrating an operation of the image processing system according to the embodiment of the present invention. [Figure 7] 7 is a diagram for supplementing the explanation of FIG. 6. [Figure 8] 7 is a diagram for supplementing the explanation of FIG. 6. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those skilled in the art to easily carry out the technical concept of the present invention.
[0012] In addition, when a part is described as being "connected" to another part throughout the specification, this includes not only the case where the part is "directly connected" to another part, but also the case where the part is "electrically connected" to another part via another element therebetween. In addition, when a part is described as "including" or "comprising" a certain component, this does not mean that the part excludes other components, but means that the part can further include or comprise other components, unless otherwise specified to the contrary. In addition, even if some components are described in the singular form throughout the specification, it will be understood that the present invention is not limited thereto, and that the component may be composed of a plurality of components.
[0013] FIG. 1 shows a block diagram of an image processing system 10 according to an embodiment of the present invention together with a subject 20.
[0014] 1, the image processing system 10 may be an electronic device such as a smartphone. For example, the image processing system 10 may include a first image sensor 100, a second image sensor 200, and an image processor 300.
[0015] The first image sensor 100 may be provided on the rear surface of the image processing system 10. For example, the first image sensor 100 may be a rear camera for photographing the rear surface of the image processing system 10. Typically, a display (not shown) of the image processing system 10 is configured on the front surface of the image processing system 10, and a user checks a desired region of interest through the display. Therefore, in the embodiment of the present invention, the first image sensor 100 is described as the rear camera, but depending on the design of the image processing system 10, the first image sensor 100 may be a camera provided on one surface of the image processing system 10 and photographs the region of interest.
[0016] The first image sensor 100 can perform at least one function on the region of interest that the user gazes at based on the setting information 3A. The first image sensor 100 can obtain a captured image IMG including the region of interest.
[0017] The second image sensor 200 may be provided on the front side of the image processing system 10. The second image sensor 200 may be a front camera for capturing an image of the front side of the image processing system 10. Typically, the display of the image processing system 10 is configured on the front side of the image processing system 10, and the user checks the area of interest through the display. Therefore, in the embodiment of the present invention, the second image sensor 200 is described as the front camera, but depending on the design of the image processing system 10, the second image sensor 200 may be a camera provided on the other side of the image processing system 10 for capturing an image of the user.
[0018] The second image sensor 200 can generate a first probability map PM1 corresponding to the gaze direction of the user's eyes (i.e., the gaze direction of the pupils) based on an information image DIMG (see FIG. 4) acquired by photographing the user. The first probability map PM1 can be a table that probabilistically represents coordinates corresponding to the positions of the pupils of the user. Furthermore, the second image sensor 200 can generate a second probability map PM2 corresponding to the angle of the user's face based on an information image DIMG acquired by photographing the user. The second probability map PM2 can be a table that probabilistically represents coordinates corresponding to the angle of the user's face. The second image sensor 200 can enter a low power mode when generating the information image DIMG and the first probability map PM1 and / or the second probability map PM2. The second image sensor 200 operates at a lower resolution and a lower frame rate in the low power mode compared to the normal mode, so that the power consumption of the second image sensor 200 may be reduced.
[0019] The image processor 300 can detect the region of interest based on the photographed image IMG and the first probability map PM1 and / or the second probability map PM2, and adjust the setting information 3A based on the region of interest. When the photographed image IMG is generated, the image processor 300 can blur the surrounding area other than the region of interest.
[0020] FIG. 2 shows a block diagram of the first image sensor 100 shown in FIG.
[0021] As shown in FIG. 2, the first image sensor 100 may include a first analog block 110, a first digital block 120, and a control block .
[0022] The first analog block 110 can generate first image signals VPX1 based on incident light entering through a rear surface of the image processing system 10. The first image signals VPX1 can each be an analog signal.
[0023] The first digital block 120 can generate a captured image IMG based on the first image signal VPX1. The captured image IMG can include a digital signal converted from the first image signal VPX1.
[0024] The control block 130 can control at least one function based on the setting information 3A. The function can include at least one of auto white balance, auto exposure, and auto focus. The setting information 3A can include a first setting value associated with the auto white balance, a second setting value associated with the auto exposure, and a third setting value associated with the auto focus.
[0025] For example, the control block 130 may include a first storage circuit 131 and a timing control circuit 133 .
[0026] The first storage circuit 131 can store and update the setting information 3A.
[0027] The timing control circuit 133 can control the at least one function based on the setting information 3 A. For example, the timing control circuit 133 can correct the color temperature of the region of interest based on the first setting value, can correct the exposure time of the region of interest based on the second setting value, and can correct the focus of the region of interest based on the third setting value.
[0028] FIG. 3 shows a block diagram of the second image sensor 200 shown in FIG.
[0029] As shown in FIG. 3, the second image sensor 200 may include a second analog block 210 and a second digital block 220 .
[0030] The second analog block 210 can generate second image signals VPX2 based on incident light entering through the front surface of the image processing system 10. The second image signals VPX2 can each be an analog signal.
[0031] The second digital block 220 can generate an information image DIMG and a first probability map PM1 and / or a second probability map PM2 based on the second image signal VPX2.
[0032] FIG. 4 shows in block diagram form the second digital block 220 shown in FIG.
[0033] As shown in FIG. 4, the second digital block 220 may include a signal conversion circuit 221, a first detection circuit 222, a first probability map generation circuit 223, a second detection circuit 224, a calculation circuit 225, a second probability map generation circuit 226, and a second storage circuit 227.
[0034] The signal conversion circuit 221 can generate an information image DIMG based on the second image signal VPX2. The information image DIMG can include a digital signal converted from the second image signal VPX2.
[0035] The first detection circuit 222 can detect the user's eye region ROE from the information image DIMG.
[0036] The first probability map generating circuit 223 can generate the first probability map PM1 based on the eye region ROE. For example, the first probability map generating circuit 223 can generate the first probability map PM1 that probabilistically represents coordinates corresponding to the gaze direction of the pupil in the eye region ROE (see FIG. 7).
[0037] The second detection circuit 224 can detect the user's face area ROF from the information image DIMG.
[0038] The calculation circuit 225 can calculate at least one rotation value VOR of the user's face based on the face region ROF, which can include at least one of a roll value of the face, a pitch value of the face, and a yaw value of the face.
[0039] The second probability map generating circuit 226 can generate a second probability map PM2 corresponding to the angle of the face of the user based on the rotation value VOR. For example, the second probability map generating circuit 226 can generate a second probability map PM2 that probabilistically represents coordinates corresponding to the angle of the face of the user.
[0040] The second storage circuit 227 may store the first probability map PM1 and / or the second probability map PM2.
[0041] In an embodiment of the present invention, the first detection circuit 222, the first probability map generation circuit 223, the second detection circuit 224, the calculation circuit 225, the second probability map generation circuit 226, and the second storage circuit 227 are described as being included in the second image sensor 200, but this is not necessarily limited to this, and some or all of the first detection circuit 222, the first probability map generation circuit 223, the second detection circuit 224, the calculation circuit 225, the second probability map generation circuit 226, and the second storage circuit 227 may also be included in the image processor 300.
[0042] FIG. 5 is a block diagram of the image processor 300 shown in FIG.
[0043] As shown in FIG. 5, the image processor 300 may include an object detection circuit 310 and a setting control circuit 320.
[0044] The object detection circuit 310 detects the region of interest based on the captured image IMG and the first probability map PM1 and / or the second probability map PM2, and detects the object OD included in the region of interest. That is, the object detection circuit 310 can detect the object OD that the user is gazing at.
[0045] The setting control circuit 320 can adjust the setting information 3A with the subject OD as the target, based on the photographed image IMG and the subject OD.
[0046] The operation of the image processing system 10 according to the embodiment of the present invention having the above-mentioned configuration will be described below with reference to Figures 6 to 8. In the following description, the first image sensor 100 will be referred to as the "rear camera" and the second image sensor 200 will be referred to as the "front camera".
[0047] FIG. 6 is a flowchart illustrating the operation of the image processing system 10 according to the embodiment of the present invention.
[0048] 6, when the rear camera 100 is enabled, the front camera 200 may be enabled (S101). At this time, the rear camera 100 may enter a normal mode and generate a captured image IMG at a preset resolution and a preset frame rate, and the front camera 200 may enter a low power mode and generate a corrected image DIMG and a first probability map PM1 and / or a second probability map PM2 at a lower resolution and a lower frame rate than in the normal mode. During the low power mode, the front camera 200 operates at the lower resolution and frame rate, thereby reducing power consumption of the front camera 200.
[0049] The front camera 200 can capture an image of a user and generate a first probability map PM1 that probabilistically represents the direction in which the user's eyes are fixed (S103). The front camera 200 can generate a second probability map PM2 that probabilistically represents the angle of the user's face (S105). If the reliability of the first probability map PM1 is high by design, the front camera 200 may not generate the second probability map PM2.
[0050] The image processor 300 can detect a region of interest that the user gazes at from the captured image IMG based on the first probability map PM1 and / or the second probability map PM2, and adjust the setting information 3A corresponding to the region of interest (S107). The setting information 3A can include a first setting value associated with auto white balance, a second setting value associated with auto exposure, and a third setting value associated with auto focus.
[0051] The rear camera 100 performs at least one function on the region of interest based on the setting information 3A to obtain a captured image IMG including the region of interest (S109). The function may include at least one of the automatic white correction, the automatic exposure, and the automatic focus.
[0052] Meanwhile, when the captured image IMG is generated, the image processor 300 can blur the surrounding area other than the region of interest.
[0053] FIG. 7 shows a diagram for explaining the first probability map PM1 described in FIG.
[0054] As shown in Fig. 7, the first probability map PM1 can probabilistically represent coordinates corresponding to the gaze direction of the user's pupil in the eye region ROE. If the gaze directions of the pupil are set to nine, the first probability map PM1 can also have nine coordinates LT, CT, RT, LC, CC, RC, LB, CB, and RB. The nine coordinates LT, CT, RT, LC, CC, RC, LB, CB, and RB can be represented as probability values or digital values (0 or 1) according to the gaze direction of the user among the nine gaze directions.
[0055] FIG. 8 shows a diagram for explaining the second probability map PM2 explained in FIG.
[0056] As shown in FIG. 8, the second probability map PM2 may probabilistically represent coordinates corresponding to the user's face angle in the face region ROF. For example, the face angle may correspond to a rotation value VOR of the user's face. The rotation value VOR may include at least one of a roll value of the face, a pitch value of the face, and a yaw value of the face. If the face angle is set to nine, the second probability map PM2 may also have nine coordinates LT, CT, RT, LC, CC, RC, LB, CB, and RB. The nine coordinates LT, CT, RT, LC, CC, RC, LB, CB, and RB may be represented as probability values or digital values (0 or 1) according to an angle corresponding to the user's gaze direction among the nine angles.
[0057] Such an embodiment of the present invention has the advantage that it is possible to automatically detect a region of interest desired by a user and automatically perform scheduled operations (e.g., white balancing operations, exposure operations, focusing operations, etc.) on the region of interest.
[0058] Although the technical concept of the present invention has been specifically described by the above embodiment, it should be noted that the above-described embodiment is for the purpose of explanation and not for the purpose of limiting the present invention. In addition, a person skilled in the art of the present invention will understand that various embodiments are possible by various substitutions, modifications, and changes within the scope of the technical concept of the present invention. [Explanation of symbols]
[0059] 10 Image Processing System 100 First image sensor 200 Second Image Sensor 300 Image Processor
Claims
1. a first detection circuit for detecting an eye region of the user from an information image of the user; a first probability map generating circuit for generating a first probability map corresponding to a gaze direction of the user's eyes based on the eye regions; a storage circuit for storing the first probability map; An image sensor comprising:
2. The image sensor of claim 1 , wherein the first probability map includes a table that probabilistically represents coordinates corresponding to positions of the user's pupils.
3. a second detection circuit for detecting a face region of the user from the information image; a calculation circuit for calculating at least one rotation value of the user's face based on the face region; a second probability map generating circuit for generating a second probability map corresponding to an angle of the face of the user based on the at least one rotation value; Further equipped with The image sensor of claim 1 , wherein the storage circuitry further stores the second probability map.
4. The image sensor of claim 3 , wherein the at least one rotation value comprises at least one of a face roll value, a face pitch value, and a face yaw value.
5. The image sensor of claim 3 , wherein the second probability map includes a table that probabilistically represents coordinates corresponding to angles of the user's face.
6. a first image sensor for performing at least one function on a region of interest that a user gazes at based on setting information, and acquiring a photographed image including the region of interest; a second image sensor for generating a first probability map corresponding to a gaze direction of the user based on an information image acquired by photographing the user; an image processor for detecting the region of interest based on the captured image and the first probability map, and adjusting the setting information based on the region of interest; An image processing system comprising:
7. The at least one function includes at least one of an auto white balance, an auto exposure, and an auto focus; The image processing system of claim 6 , wherein the setting information includes at least one of a setting value associated with the automatic white correction, a setting value associated with the automatic exposure, and a setting value associated with the automatic focus.
8. The image sensor of claim 6 , wherein the first probability map includes a table that probabilistically represents coordinates corresponding to the positions of the user's pupils.
9. The first image sensor includes: a first analog block for generating a first image signal based on incident light entering through a rear face of the image processing system; a first digital block for generating the captured image based on the first image signal; a control block for controlling the at least one function based on the configuration information; 7. The image processing system of claim 6, comprising:
10. The control block comprises: a first storage circuit for storing and updating the configuration information; a timing control circuit for controlling the at least one function based on the setting information; 10. The image processing system of claim 9, comprising:
11. 7. The image processing system of claim 6, wherein the second image sensor captures the information image in a low power mode and generates the first probability map.
12. The second image sensor further generates a second probability map corresponding to an angle of the user's face based on the information image of the user; 7. The image processing system of claim 6, wherein the image processor further uses the second probability map when detecting the region of interest.
13. The second image sensor is a second analog block for generating a second image signal based on incident light entering through a front surface of the image processing system; a second digital block for generating the information image and the first probability map based on the second image signal; 7. The image processing system of claim 6, comprising:
14. The second digital block comprises: a first detection circuit for detecting an eye region of the user from the information image; a first probability map generating circuit for generating the first probability map based on the eye regions; a second storage circuit for storing the first probability map; 14. The image processing system of claim 13, comprising:
15. The second digital block comprises: a second detection circuit for detecting a face area of the user from the information image of the user; a calculation circuit for calculating at least one rotation value of the user's face based on the face region; a second probability map generating circuit for generating a second probability map corresponding to an angle of the face of the user based on the at least one rotation value; The image processing system of claim 14 further comprising:
16. 16. The image processing system of claim 15, wherein the at least one rotation value comprises at least one of a facial roll value, a facial pitch value, and a facial yaw value.
17. 16. The image processing system of claim 15, wherein the second probability map comprises a table of probabilistic coordinates corresponding to angles of the user's face.
18. The image processing system according to claim 6 , wherein the image processor blurs surrounding areas other than the area of interest when the captured image is generated.
19. The image processor includes: an object detection circuit for detecting an object included in the region of interest based on the captured image and the first probability map; a setting control circuit for adjusting the setting information for the subject based on the photographic image and the subject; 7. The image processing system of claim 6, comprising:
20. if the rear camera is enabled and enters a normal mode, the front camera is enabled and enters a low power mode; said front camera photographing a user and generating a first probability map representative of a direction in which the user's eyes are gazed; an image processor detecting a region of interest where the user's eyes are fixed based on the first probability map, and adjusting setting information corresponding to the region of interest; The rear camera performs at least one function on the region of interest based on the setting information, and captures a photographed image including the region of interest; A method of operating an image processing system comprising:
21. The method further includes generating a second probability map corresponding to an angle of the user's face based on an information image captured by the front camera of the user; 21. The method of claim 20, wherein the image processor adjusting the configuration information further comprises using the second probability map when detecting the region of interest.
22. The method of claim 20, wherein the image processor blurs surrounding areas other than the area of interest when the captured image is generated.
Citation Information
Cited By
Electronic Device with Centrally Located Under-Display Image Sensor
US20250131534A1
