Electronic apparatus and image processing method

Abstract

To keep quality of a captured image while continuing screen display.SOLUTION: Light transmitted by a transmission area being a partial area of a display with a plurality of pixels aligned therein is incident on a camera. The controller generates a difference image between a first image acquired when display information is displayed in the transmission area and a second image acquired when the display information is not displayed, and generates a restored image being a difference between a third image captured when the display information is displayed in the transmission area and the difference image.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present application relates to an electronic device and an image processing method, for example an electronic device having a display and a camera. [Background technology]

[0002] Some electronic devices, such as clamshell-type PCs (Personal Computers), have a display and a camera in the case. To avoid interfering with the screen display as much as possible, an opening is provided in the upper center of the screen area, and the camera is sometimes installed in the opening. Since the opening is provided outside the screen area, the screen occupancy rate is limited. In recent years, PCs with a high screen occupancy rate have become popular, and PCs that use the CUD (Camera Under Display) method are becoming more and more practical.

[0003] In recent years, in response to demands for devices with a high screen occupancy rate, electronic devices that adopt the CUD (Camera Under Display) method have been proposed. The CUD method is configured such that a camera is installed on the back side of a transparent area that forms part of the screen area. The camera captures an image that appears in the light that has passed through the transparent area. For example, Patent Document 1 describes a display device that covers the device body, has an element area positioned behind the display area, and includes a photosensitive element in the element area that collects light passing through the display area. The photosensitive element includes a camera.

[0004] The CUD method is a configuration in which a camera is installed on the back side of the screen area of ​​a display. The camera captures an image that appears in light that has passed through the screen area. For example, Patent Document 1 describes a display device that covers the device body, has an element area located behind the display area, and includes a photosensitive element in the element area that collects light that passes through the display area. The photosensitive element includes a camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2022-513506 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the CUD method, when capturing an image with a camera, if the screen display is maintained in a transparent area of ​​the screen area that covers the optical system of the camera, the image quality of the captured image may be significantly lower than when the screen display is not performed. In the transparent area where the screen display is performed, the pixels arranged in that area are lit up. Light emitted from the pixels enters the optical system of the camera. Incident light from the subject surface and incident light from the pixels arrive at the imaging surface of the imaging element. The captured image captured when the screen display is performed may have lower overall contrast and locally different brightness and color than when the screen display is not performed. [Means for solving the problem]

[0007] The present application has been made to solve the above-mentioned problems, and an electronic device according to one aspect of the present application includes a display on which a plurality of pixels are arranged, a camera, and a controller, wherein light that has passed through a transparent region that is a portion of the display is incident on the camera, and the controller generates a difference image between a first image captured when display information is displayed in the transparent region and a second image captured when the display information is not displayed, and generates a restored image that is the difference between the difference image and a third image captured when display information is displayed in the transparent region.

[0008] In the above electronic device, the controller may obtain the first image and the second image from the camera when the display information is still information.

[0009] The electronic device may further include a detection unit that detects a person in the vicinity of the electronic device, and when the person is not detected, the controller may obtain the first image and the second image.

[0010] In the above-mentioned electronic device, an influence map, which is an image captured by emitting light at a constant brightness for each light-emitting unit arranged in the transparent area, may be set in advance, and the controller may accumulate the influence map based on a signal value representing the display information for each light-emitting unit to generate the difference image.

[0011] In the electronic device described above, the controller may estimate the difference image captured using an imaging parameter of the third image based on the imaging parameter.

[0012] In the above electronic device, the influence map may be set in advance for each imaging parameter, and the controller may identify the influence map corresponding to the imaging parameter of the third image, and generate the difference image using the identified influence map.

[0013] In the electronic device described above, the controller may reduce a luminance of the display information when a component ratio of the difference image in the third image is greater than a predetermined reference value of the component ratio.

[0014] An image processing method according to a second aspect of the present application is an image processing method for an electronic device comprising a display on which a plurality of pixels are arranged, a camera, and a controller, in which light that has passed through a transparent area that is a portion of the display is incident on the camera, wherein the electronic device generates a difference image between a first image acquired when display information is displayed in the transparent area and a second image acquired when the display information is not displayed, and generates a restored image that is the difference between the difference image and a third image captured when display information is displayed in the transparent area. Effect of the Invention

[0015] According to this embodiment, it is possible to maintain the image quality of a captured image while continuing to display the screen. [Brief description of the drawings]

[0016] [Figure 1]1 is a front view showing an example of the external configuration of an information processing device according to a first embodiment. [Diagram 2] 1 is a cross-sectional view illustrating a cross section of an information processing device according to a first embodiment. [Diagram 3] 1 is a schematic block diagram illustrating an example of a hardware configuration of an information processing device according to a first embodiment. [Figure 4] 1 is a schematic block diagram illustrating an example of a functional configuration of an information processing device according to a first embodiment. [Diagram 5] FIG. 4 is an explanatory diagram showing an example of generation of a difference image according to the first embodiment. [Figure 6] FIG. 4 is an explanatory diagram showing an example of generation of a restored image according to the first embodiment. [Figure 7] 4 is a flowchart illustrating captured image processing according to the first embodiment. [Figure 8] 1 is a schematic block diagram illustrating an example of a functional configuration of an information processing device according to a first embodiment. [Figure 9] FIG. 11 is an explanatory diagram showing an example of obtaining an influence map according to the second embodiment; [Figure 10] 10 is a flowchart illustrating captured image processing according to the second embodiment. [Figure 11] 13 is a flowchart showing an example of captured image processing according to the third embodiment. [Figure 12] 13 is a flowchart showing another example of captured image processing according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] <First embodiment> Hereinafter, an electronic device according to an embodiment of the present application will be described with reference to the drawings. In the following description, the electronic device will be mainly an information processing device 10 configured as a clamshell type PC. A clamshell type PC is also called a notebook type PC or a notebook PC. First, an information processing device 10 according to a first embodiment will be described. FIG. 1 is a front view showing an example of the external configuration of the information processing device 10 according to this embodiment. FIG. 2 is a cross-sectional view illustrating a cross section of the information processing device 10 according to this embodiment.

[0018] The information processing device 10 includes a first housing 10a and a second housing 10b. The first housing 10a is rotatable relative to the second housing 10b, and the angle between the surface of the first housing 10a and the surface of the second housing 10b (sometimes referred to as the "opening / closing angle" in this application) is variable. The long side of the first housing 10a and the long side of the second housing 10b are connected using hinge mechanisms 121a and 121b. The hinge mechanisms 121a and 121b enable the first housing 10a to rotate relatively to the second housing 10b around a rotation axis ax. The direction of the rotation axis ax is parallel to both the long side of the first housing 10a and the long side of the second housing 10b. The hinge mechanisms 121a and 121b can maintain an arbitrary opening / closing angle θ even when some torque is applied.

[0019] The first housing 10a is formed in a box shape. A display 14 and a camera 28 are mounted on the surface of the first housing 10a. A keyboard 107 and a touchpad 109 are mounted inside the second housing 10b and on the surface of the second housing 10b. When the first housing 10a is opened relative to the second housing 10b, a user facing the front of the first housing 10a can see the display information displayed on the display 14 and can perform input operations on the keyboard 107 and the touchpad 109. At this time, the camera 28 can capture an image of the user's head. In the following description, the first housing 10a and the second housing 10b may be collectively referred to simply as "housings".

[0020] The display 14 has a substantially flat plate shape and is configured as a display panel. The display 14 covers most of the surface of the first housing 10a, and its periphery is supported by the first housing 10a. The display 14 includes a substrate 14b. A plurality of pixels 14p are arranged two-dimensionally at regular intervals on the substrate 14b. The region in which the plurality of pixels 14p are arranged forms a screen region SA. Display information is expressed by the brightness or color distribution in the screen region SA. In the transmissive region TA, which is a part of the screen region SA, the pixels 14p are arranged more sparsely than in the normal region NA, which is the surrounding region. In the transmissive region TA, the adjacent pixels 14p are arranged with gaps between them without being in complete contact with each other. Light incident on the transmissive region TA passes through the gaps between the pixels and enters the optical system of the camera 28. The pixel interval in the transmissive region TA may be equal to the pixel interval in the normal region NA. In that case, the size of the pixels arranged in the transmissive region TA may be smaller than the size of the pixels arranged in the normal region NA.

[0021] As described above, the information processing device 10 includes the display 14 and the camera 28. The display 14 has a screen area in which a plurality of pixels are arranged. Light transmitted through the transparent area TA is incident on the camera 28. The transparent area TA forms a part of the screen area SA of the display 14. The information processing device 10 generates a difference image between a first image acquired when display information is presented in the transparent area TA and a second image acquired without the display information being presented. The information processing device 10 generates a restored image that is a difference between a third image captured when display information is presented in the transparent area TA and the difference image. That is, the restored image is obtained by removing the influence of the display information displayed on the display 14 from the third image.

[0022] Next, a hardware configuration example of the information processing device 10 according to the present embodiment will be described. Fig. 3 is a schematic block diagram showing a hardware configuration example of the information processing device 10 according to the present embodiment. The information processing device 10 includes a processor 11, a main memory 12, a video subsystem 13, a display 14, a chipset 21, a ROM 22, an auxiliary storage device 23, an audio system 24, a communication module 25, an input / output interface 26, a camera 28, an EC 31, an input device 32, a power supply circuit 33, and a battery 34.

[0023] The processor 11 is a core processing device that executes various arithmetic processing instructed by commands written in software (programs). The processing executed by the processor 11 includes reading and writing data from and to storage media such as the main memory 12 and the auxiliary storage device 23, and input and output with other devices. The processor 11 includes at least one CPU. The CPU controls the operation of the entire information processing device 10. The CPU executes processing based on programs such as an OS (Operating System), firmware, device drivers, utilities, and application programs (also referred to as "apps" in this application). Note that in this application, executing processing instructed by commands written in various programs may be referred to as "executing a program" or "running a program", etc.

[0024] The main memory 12 is a writable memory used as a read area for the execution program of the processor 11 or as a work area for writing processing data for the execution program. The main memory 12 is composed of, for example, a plurality of DRAM (Dynamic Random Access Memory) chips. The processor 11 and the main memory 12 correspond to the minimum hardware constituting the host system 100 (described later). The host system 100 is a computer system that forms the core of the information processing device 10.

[0025] Video subsystem 13 is a subsystem for realizing functions related to image display, and includes a video controller. The video controller processes drawing commands from processor 11, writes the obtained drawing information to a video memory, and also reads the drawing information from the video memory and outputs it to display 14 as display data indicating display information (image processing).

[0026] The display 14 displays a display screen based on the display data input from the video subsystem 13. The display 14 may be, for example, any of a liquid crystal display (LCD) and an organic light emitting diode (OLED) display.

[0027] The chipset 21 includes a plurality of controllers and can be connected to a plurality of devices to input and output various data. The controller is, for example, any one or a combination of bus controllers such as a Universal Serial Bus (USB), a serial AT Attachment (Serial ATA), a Serial Peripheral Interface (SPI) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express bus, and a Low Pin Count (LPC). The plurality of devices include, for example, a ROM 22, an auxiliary storage device 23, an audio system 24, a communication module 25, an input / output interface 26, and an EC 31.

[0028] A ROM (Read Only Memory) 22 mainly stores system firmware, firmware for controlling the operation of the EC 31 and other devices, etc. The ROM 22 may be, for example, any of an EEPROM (Electrically Erasable Programmable Read Only Memory) and a flash ROM.

[0029] The auxiliary storage device 23 stores various data used in processing by the processor 11 and other devices, or various data acquired by such processing, various programs, etc. The auxiliary storage device 23 may be, for example, any one of an SSD (Solid State Drive), an HDD (Hard Disk Drive), etc., or a combination of any of them.

[0030] The audio system 24 is connected to a microphone and a speaker (not shown) and records, plays, and outputs audio data. The microphone and the speaker may be built into the information processing device 10 or may be separate from the information processing device 10.

[0031] The communication module 25 is connected to a communication network wirelessly or by wire. The communication module 25 communicates various data with other devices connected to the communication network. The communication module includes, for example, a wireless LAN (Local Area Network), and enables various data to be transmitted and received between devices according to a predetermined wireless communication method (for example, IEEE802.11).

[0032] The input / output interface 26 is connected to various devices including peripherals via wired or wireless connection, and is, for example, a connector for inputting and outputting data via wired connection in accordance with the USB standard.

[0033] The camera 28 captures an image including an image of a subject located within the field of view. The camera 28 outputs captured image data indicating the captured image obtained by capturing the image to the processor 11 via the chip set 21.

[0034] The EC (Embedded Controller) 31 is a one-chip microcomputer that monitors and controls various devices (peripheral devices, sensors, etc.) regardless of the operating state of the system of the information processing device 10. The EC 31 includes a CPU, a ROM, a RAM, an A / D (Analog-to-Digital) input terminal for multiple channels, a D / A (Digital-to-Analog) output terminal, a timer, and digital input / output terminals (not shown) separate from the processor 11. For example, an input device 32, a power supply circuit 33, etc. are connected to the input / output terminals of the EC 31.

[0035] The input device 32 detects a user's operation and outputs an operation signal corresponding to the detected operation to the EC 31. The above-mentioned keyboard 107 and touch pad 109 correspond to the input device 32. The input device 32 may be a touch sensor, or may overlap the display 14 and be configured as a touch panel.

[0036] The power supply circuit 33 converts the voltage of DC power supplied from an external power supply or a battery 34 to a voltage required for the operation of each device constituting the information processing device 10, and supplies the power having the converted voltage to the device as a supply destination. The power supply circuit 33 executes power supply according to the control of the EC 31. The power supply circuit 33 includes a converter that converts the voltage of the power supplied to itself, and a power supply device that charges the battery 34 with the power whose voltage has been converted. The power supply device charges the battery 34 with the power that is supplied from the external power supply and that is not consumed by each device. When power is not supplied from the external power supply, or when the power supplied from the external power supply is insufficient, the power discharged from the battery 34 is supplied to each device as operating power. The battery 34 charges or discharges power using the power supply circuit 33. The battery 34 may be, for example, a lithium ion battery, a sodium ion battery, or the like.

[0037] Next, an example of the functional configuration of the information processing device 10 according to this embodiment will be described. Fig. 4 is a schematic block diagram showing an example of the functional configuration of the information processing device 10 according to this embodiment. The information processing device 10 includes a camera 28, a host system 100, a display 14, and a storage unit 120. Some or all of the functions of the host system 100 are realized when the processor 11 executes an application program (sometimes referred to as an "app" in this application) and other programs on the OS, in cooperation with the main memory 12, the camera 28, the display 14, the storage unit 120, and other hardware.

[0038] The host system 100 includes a display control unit 102, a difference image generation unit 104, an image restoration unit 106, and an output processing unit . The display control unit 102 controls the display of display information based on the display data on the display 14. The display control unit 102 controls, for example, brightness, contrast, whether display is required (ON / OFF), and the like, according to an operation signal input from, for example, the input device 32 or the input / output interface 26. The display information can be acquired by executing other programs (including apps). The display information corresponds to any one or any combination of characters, figures, symbols, and patterns. The display information is configured as a display screen, and can be displayed across a screen area SA that typically includes a transparent area TA and a normal area NA. The display information may include moving images, or may include still information without moving images.

[0039] The differential image generating unit 104 acquires captured image data indicating an image captured by the camera 28 when the display information is displayed in the transparent area TA. In the present application, the image acquired when the display information is displayed may be referred to as a "first image". Thereafter, the differential image generating unit 104 causes the display 14 to stop displaying at least the display information in the transparent area TA. The differential image generating unit 104 acquires captured image data indicating an image captured by the camera 28 when the display information is not displayed in the transparent area TA. In the present application, the image acquired when the display information is not displayed may be referred to as a "second image". However, the differential image generating unit 104 acquires a second image representing an image of a common subject in a common field of view with the first image. Thereafter, the differential image generating unit 104 may cause the display 14 to resume displaying the display information.

[0040] The difference image generating unit 104 generates a difference between the first image and the second image as a difference image. When generating the difference image, the difference image generating unit 104 calculates a difference value between a signal value indicating the first image and a signal value indicating the second image for each pixel. The difference image generating unit 104 generates image data indicating the difference value calculated for each pixel as difference image data, and stores the generated difference image data in the storage unit 120. Therefore, a component caused by light emission of the pixels arranged in the transmission area TA in the first image is extracted as the difference image.

[0041] Next, an example of generating a difference image will be described. FIG. 5 is an explanatory diagram showing an example of generating a difference image. FIG. 5 shows a first image, a second image, and a difference image in that order on the left and right. Both the first image and the second image show a plain background in the field of view and a star in the center as a subject. However, the first image is brighter overall than the second image, and has a lower contrast. In particular, the brightness difference between the two images is more significant in the region from the center to the bottom than in other regions. The image of the subject does not appear in the difference image, and the region from the center to the bottom has a brighter background than other regions.

[0042] Returning to FIG. 4, the image restoration unit 106 will be described. When the display information is displayed in the transparent area TA, the image restoration unit 106 acquires captured image data indicating the captured image captured by the camera 28. In the present application, the image acquired by the image restoration unit 106 when the display information is displayed may be referred to as the "third image." In general, the third image is acquired at a later time than the first image and the second image. The first image may be acquired either before or after the second image. However, it is desirable that the display information displayed in the transparent area TA is the same as the display information at the time of capturing the first image and the second image.

[0043] The image restoration unit 106 refers to the differential image signal from the storage unit 120 and generates a restored image that is a difference between the third image and the differential image. When generating the restored image, the image restoration unit 106 calculates a difference value between the signal value of the third image and the signal value of the differential image for each pixel. The image restoration unit 106 generates image data indicating the difference value calculated for each pixel as restored image data indicating the restored image, and outputs the image data to the output processing unit 108.

[0044] Next, an example of generating a restored image will be described. FIG. 6 is an explanatory diagram showing an example of generating a restored image. FIG. 6 shows a third image, a difference image, and a restored image in that order on the left and right. When display information is displayed in the transparent area TA, the third image shows a person's head as a subject different from the first image or the second image. In the background of the third image, the area from the center to the bottom is brighter than the rest. The difference image exemplified in FIG. 6 is the same image as the difference image exemplified in FIG. 5. The restored image shows a subject common to the third image, and the luminance of the background is spatially uniform and is darker overall than the luminance of the background of the third image. The phenomenon in which the luminance of the area from the restored center to the bottom in the third image is higher than that of the surrounding areas is eliminated by subtracting the difference image.

[0045] Returning to Fig. 4, the output processing unit 108 will be described. The output processing unit 108 performs processing related to the output of restored image data acquired from the image restoration unit 106. The output processing unit 108 may output the restored image data to another device via the display 14 or the input / output interface 26, or may store the restored image data in the storage unit 120. The output destination of the restored image data may be specified by an operation signal input from the input device 32 or the input / output interface 26, or may be specified by a control signal generated by processing of another application.

[0046] Next, an example of captured image processing according to this embodiment will be described with reference to Fig. 7, which is a flowchart illustrating the captured image processing according to this embodiment. (Step S102) The display control unit 102 acquires display data, and causes the display 14 to display the display information indicated in the display data. (Step S104) While display information is being displayed on the display 14, the differential image generating unit 104 causes the camera 28 to capture a first image, and acquires the first image from the camera 28.

[0047] (Step S106) The difference image generating unit 104 temporarily stops displaying the display information on the display 14. (Step S108) When no display information is displayed on the display 14, the difference image generating unit 104 causes the camera 28 to capture a second image, and acquires the second image from the camera 28. (Step S110) The difference image generating unit 104 generates a difference image between the first image and the second image.

[0048] (Step S112) The difference image generating unit 104 causes the display 14 to resume displaying the display information. (Step S114) While displaying the display information on the display 14, the image restoration unit 106 causes the camera 28 to capture a third image, and acquires the third image from the camera 28. (Step S116) The image restoration unit 106 generates a restored image which is the difference between the third image and the difference image. After that, the process in FIG.

[0049] In the process of FIG. 7, after the difference image is generated in step S110, the processes of steps S102 to S112 may be omitted and the processes of steps S114 and S116 may be repeated while the display information is maintained unchanged. In addition, in step S104, the differential image generating unit 104 may monitor the display data output from the display control unit 102 to the display 14, and determine whether or not the display information is being displayed in at least the transmissive area TA of the display 14. Similarly, in step S114, the image restoration unit 106 can determine, based on the display data output from the display control unit 102, whether or not the display information is being displayed at least in the transmissive area TA.

[0050] In step S114, the image restoration unit 106 may determine whether or not the display information notified in the display data from the display control unit 102 in this step is the same as the display information notified in the display data from the display control unit 102 in step S104. If the image restoration unit 106 determines that they are the same, it may proceed to capturing a third image, and if it does not determine that they are the same, it may end the processing in FIG. 7 without proceeding to capturing a third image.

[0051] When displaying a still image on the display 14 as the display information, the display control unit 102 may start executing the process of Fig. 7. As a result, in step S114, the same display information as that in step S104 is displayed, and the possibility of successfully completing the process of Fig. 7 can be increased. When the display of the still image ends, the display control unit 102 may end the process of Fig. 7. Ending the display of the still image may be instructed, for example, by an operation signal indicating screen erasure. 7 every time a new still image is displayed on the display 14. In this way, a restored image is obtained from the third image by using a difference image that shows a different effect for each still image.

[0052] The differential image generating unit 104 may execute the processes of steps S104 to S112 when the user of the information processing device 10 is not facing the display 14. This reduces the possibility that the user feels uncomfortable because the user notices the switching between display and non-display of the display information. In addition, this reduces the possibility that the image of the subject changes between the first image and the second image because the user moves or appears and disappears while facing the display 14.

[0053] Therefore, before proceeding to the process of step S104, the display control unit 102 may display guidance information on the display 14 to guide the user to move out of the field of view of the camera 28 at least until the process of step S112 is completed. The guidance information includes, for example, a message such as "We will now set up the camera at the bottom of the screen. Please move to the left or right side of the display 14 so that you are not captured by the camera until a message indicating that the settings are complete appears." When the process of step S112 is completed, the display control unit 102 displays the message "Settings Complete" on the display 14. The display area of ​​the guidance information may or may not include the transparent area TA.

[0054] The information processing device 10 may also include a detection unit that detects a person around the device. The information processing device 10 may include a human presence sensor (not shown) as an example of a detection unit, and when detecting the presence of a person within a certain detection range from the sensor, may output a detection signal indicating the location of the person to the display control unit 102. The information processing device 10 may include an image recognition unit (not shown) that includes image recognition processing instead of or together with the human presence sensor. The image recognition unit executes a known image recognition process on a captured image shown in the captured image data input from the camera 28, and determines whether or not a person can be detected as a subject appearing in the captured image. When detecting a person, the image recognition unit outputs a detection signal indicating the location of the person to the display control unit 102.

[0055] When a detection signal is input from the human presence sensor or the image recognition unit, the display control unit 102 does not execute the process of Fig. 7, and when the detection signal is not input, executes the process of Fig. 7. However, after the differential image is generated, when the display information is not changed and is kept constant, the processes of steps S114 and S116 may be executed using the differential image related to the display information.

[0056] <Second embodiment> Next, a second embodiment of the present invention will be described. The description of this embodiment will mainly focus on the differences from the first embodiment, and the description of the first embodiment will be used for the points in common unless otherwise specified.

[0057] 8 is a schematic block diagram showing an example of a functional configuration of the information processing device 10 according to this embodiment. The host system 100 of the information processing device 10 includes a display control unit 102, a difference image generating unit 104, an image restoring unit 106, and an output processing unit 108, as well as an influence map setting unit 110.

[0058] When display information is not displayed on the display 14, the influence map setting unit 110 acquires an image obtained under certain imaging conditions by making each pixel arranged in the transparent region TA emit light at a certain luminance as an influence map. The imaging conditions are specified by imaging parameters. The imaging parameters include, for example, an exposure value (EV). A predetermined exposure value may be set in advance in the influence map setting unit 110. The exposure value indicates a combination of an F-number and an exposure time. The F-number corresponds to the ratio of the focal length of the objective lens to the diameter of the aperture of the optical system. The F-number indicates that the larger the value is, the less the amount of incident light is, and the darker the captured image is. The influence map can also be regarded as an image expressing the degree of influence of each pixel on the captured image. The influence map setting unit 110 stores influence map data indicating the acquired influence map in the storage unit 120.

[0059] 9, the influence map setting unit 110 causes the camera 28 to capture an influence map while each pixel arranged in the transparent region TA is being illuminated. The influence map setting unit 110 acquires influence map data indicating the influence map captured for each pixel from the camera 28, and stores the acquired influence map data in the storage unit 120. The influence map setting unit 110 acquires the influence map data at the latest before the differential image generating unit 104 generates the differential image, and stores the data in the storage unit 120. The timing of acquiring the influence map data may be within the period from after the production of the information processing device 10 to before its shipment, for example.

[0060] The difference image generating unit 104 refers to the influence map data generated in advance and estimates a difference image based on the display information displayed on the display 14. More specifically, the difference image generating unit 104 acquires the display information displayed on the display 14 from the display control unit 102. When acquiring the display information, the difference image generating unit 104 can, for example, call an API (Application Program Interface) function (e.g., DesktopCapture API, etc.) defined in an OS, a browser, etc., and acquire the display information as a return value that is the execution result. The estimated difference image corresponds to the difference between a first image captured while the display information is being displayed and a second image captured without displaying the display information.

[0061] The difference image generating unit 104 synthesizes an image represented by a weighted sum based on the display information of the influence map for each pixel arranged in the transparent area TA as a difference image. More specifically, the difference image generating unit 104 calculates a weighted distribution for each pixel arranged in the transparent area TA by multiplying a weight value corresponding to a signal value of the display information related to the pixel by a signal value distribution in a screen area representing the influence map related to the pixel. The signal value distribution is represented by a signal value for each pixel. The difference image generating unit 104 generates difference image data indicating the sum of the weighted distribution calculated for each pixel in the transparent area TA between pixels in the transparent area TA as a signal value distribution. The difference image generating unit 104 outputs difference image data indicating the generated difference image to the image restoring unit 106.

[0062] The image restoration unit 106 receives differential image data from the differential image generation unit 104. The image restoration unit 106 generates a restored image that is a difference between the third image captured by the camera 28 and the differential image. The differential image estimated in the differential image generation unit 104 corresponds to a difference between a first image captured while display information is being displayed and a second image captured without displaying the display information. In this embodiment, the influence of the display information on the third image is removed by subtracting, from the third image, differential information estimated based on the display information using the influence map data, without capturing the first and second images.

[0063] Next, an example of the captured image processing according to this embodiment will be described. Fig. 10 is a flowchart illustrating the captured image processing according to this embodiment. The processing in Fig. 10 is based on the premise that the influence map data is acquired in advance, the display information is displayed on the display 14, an image capturing command is issued to the camera 28, and the camera 28 starts capturing images.

[0064] (Step S122) The camera 28 captures an image, and outputs captured image data indicating the captured image to the host system 100. (Step S124) The difference image generating unit 104 acquires display information from the display control unit 102. After that, the process proceeds to loop R01.

[0065] (Loop R01) The difference image generating unit 104 initializes the signal value for each pixel of the difference image, and executes the process of step S126 for each pixel arranged in the transparent region TA. (Step S126) The difference image generating unit 104 refers to the influence map data stored in the storage unit 120, and updates the signal values ​​of the difference image by adding a multiplied value obtained by multiplying each signal value shown in the influence map related to the pixel by a weight value corresponding to the signal value of the display information related to the pixel. As the difference image, the influence map for each pixel is sequentially accumulated with a weight corresponding to the signal value of the display information of the pixel. When the process of step S126 is completed for all pixels arranged in the transparent region TA, the difference image generation unit 104 ends the process of loop R01. The difference image generation unit 104 outputs difference image data indicating the difference image to the image restoration unit .

[0066] (Step S128) The image restoration unit 106 receives the captured image data from the camera 28 and the differential image data from the differential image generation unit 104. The image restoration unit 106 generates a restored image that is the difference between the captured image indicated in the captured image data and the difference image indicated in the difference image data, and then ends the process of FIG.

[0067] This embodiment can be applied even when the display 14 is a color display. In a color display, the color that can be emitted is set in advance for each pixel, which is a light-emitting unit generally arranged in the transmissive area TA. As an arrangement of colors for each pixel, for example, a stripe arrangement can be applied. The stripe arrangement is a type of arrangement pattern in a screen area of ​​red pixels capable of emitting red light (R), green pixels capable of emitting green light (G), and blue pixels capable of emitting blue light (B). This embodiment may be applied under such a pixel arrangement to obtain an influence map and synthesize a difference image based on the obtained influence map.

[0068] In the above description, influence map data is set for each pixel, which is an emission unit arranged in the transmissive region TA, and a difference image is generated based on a weight value corresponding to a signal value of the display information for each pixel using the set influence map data, but this is not limited to the above. The emission unit, which is a component of the display information, may be a sub-pixel, which is a unit smaller than a pixel. That is, the present invention may be applied to a case where each pixel has a plurality of sub-pixels. Each pixel has, for example, three sub-pixels. The three sub-pixels are capable of emitting red light (R), green light (G), and blue light (B), respectively.

[0069] When each pixel has multiple sub-pixels, the influence map setting unit 110 causes each sub-pixel to emit light at a constant luminance, and causes the camera 28 to capture the influence map according to constant imaging conditions. The influence map data indicates the influence map for each sub-pixel. The differential image processing unit 104 uses a weighting value corresponding to the signal value for each sub-pixel constituting the display information to calculate a multiplication value by the signal value indicating the influence map corresponding to that sub-pixel, and can generate a sum of the calculated multiplication values ​​across sub-pixels in the transparent region TA as a differential image.

[0070] When multiple types of pixels or sub-pixels with different luminous colors are used in the display 14, the applied imaging parameters may include a white balance value. The white balance value is expressed as a ratio of signal values ​​between multiple types of colors. The difference image generating unit 104 may adjust the luminance ratio of the influence map between pixels or sub-pixels so as to obtain a color tone equivalent to that of the influence map that can be captured if the imaging parameters used to capture the third image are used.

[0071] Note that the imaging parameters used when capturing the third image may be different from the imaging parameters used when capturing the influence map. Therefore, the difference image generating unit 104 may acquire the imaging parameters used when capturing the third image from the camera 28, and estimate the difference image captured using the imaging parameters based on the acquired imaging parameters. More specifically, the difference image generating unit 104 adjusts the luminance of the influence map for each pixel so as to obtain a brightness equivalent to that of the influence map that can be captured if the imaging parameters acquired from the camera 28 are used.

[0072] For example, correction data indicating the amount of luminance correction for each imaging parameter is set in advance in the difference image generating unit 104. The difference image generating unit 104 can determine the amount of correction corresponding to the imaging parameter by referring to the correction data, and can determine the multiplied value obtained by multiplying the determined correction amount by the signal value of the influence map for each pixel as the signal value of the influence map after the luminance adjustment. The difference image generating unit 104 can execute the above procedure to generate a difference image with the luminance adjusted based on the display information and the adjusted influence map.

[0073] The influence map setting unit 110 may generate influence map data for each imaging parameter, and store the generated influence map data in advance in the storage unit 120. The difference image generating unit 104 acquires the imaging parameters used in capturing the third image from the camera 28, and selects influence map data corresponding to the acquired imaging parameters from the influence map data stored in the storage unit 120. The difference image generating unit 104 can generate a difference image according to the above procedure using the selected influence map data. In this case, the difference image generating unit 104 does not need to adjust the brightness of the influence map based on the imaging parameters.

[0074] <Third embodiment> Next, a third embodiment of the present invention will be described. The description of this embodiment will mainly focus on the differences from the first embodiment, and the description of the first embodiment will be used for the points in common unless otherwise specified.

[0075] In the information processing device 10 according to this embodiment, the display control unit 102 acquires captured image data indicating a third image captured by the camera 28, and acquires differential image data indicating a generated differential image from the differential image generating unit 104. The display control unit 102 calculates a component ratio for the differential image in the third image based on the acquired captured image data and differential image data. For example, the display control unit 102 calculates the sum of squares of the signal values ​​for each pixel of the third image as the power of the third image, and calculates the sum of squares of the signal values ​​for each pixel of the differential image as the power of the differential image. Then, the display control unit 102 can calculate the component ratio by dividing the power of the differential image by the power of the third image.

[0076] When the calculated component ratio is greater than a predetermined reference value of the component ratio (for example, −6 dB to −20 dB), the display control unit 102 may reduce the luminance of the display information to be displayed on the display 14. The larger the calculated component ratio and the reference value are, the smaller the gain of the drive signal supplied to the display 14 is reduced by the display control unit 102. When the influence of the display information included in the third image is high, the influence of the display information included in the third image can be reduced by reducing the luminance of the display information.

[0077] When the calculated component ratio is greater than a predetermined reference value of the component ratio, the display control unit 102 may output a restoration stop command to the image restoration unit 106, which indicates the stop of generation of the restored image. When the restoration stop command is input from the display control unit 102, the image restoration unit 106 stops the generation of the restored image. When the calculated component ratio is equal to or less than the predetermined reference value of the component ratio, the display control unit 102 may output a restoration command to the image restoration unit 106, which indicates the generation of the restored image. When the restoration command is input from the display control unit 102, the image restoration unit 106 may start the generation of the restored image. Thereby, when the influence of the display information on the third image is significant, the generation of the restored image is prevented.

[0078] Next, an example of captured image processing according to this embodiment will be described. Fig. 11 is a flowchart showing an example of captured image processing according to this embodiment. The processing in Fig. 11 further includes steps S132, S134, and S136 in addition to the processing in Fig. 7. After the processing in step S114 is completed, the processing proceeds to step S132.

[0079] (Step S132) The display control unit 102 acquires the captured image data from the camera 28 and the difference image data from the difference image generating unit 104, and calculates a component ratio for the difference image in the third image. (Step S134) The display control unit 102 determines whether the calculated component ratio is greater than a predetermined reference value. If it is determined that the calculated component ratio is greater than the reference value (YES in step S134), the process proceeds to step S136. If it is determined that the calculated component ratio is equal to or less than the reference value (NO in step S134), the process proceeds to step S116. (Step S136) The display control unit 102 reduces the luminance of the display information to be displayed on the display 14. After that, the process proceeds to step S104.

[0080] Fig. 11 illustrates an example in which the luminance adjustment of the display information based on the component ratio by the display control unit 102 is applied to the process of Fig. 7, but is not limited thereto. As illustrated in Fig. 12, the luminance adjustment of the display information based on the component ratio by the display control unit 102 may be applied to the process of Fig. 10. The process of Fig. 12 further includes processes of steps S132, S134, and S136 in addition to the process of Fig. 10.

[0081] After the process of loop R01 is completed, the process proceeds to step S132. If it is determined in step S134 that the value is greater than the reference value (YES in step S134), the process proceeds to step S136. After the process of step S136 is completed, the process returns to step S122. If it is determined in step S134 that the value is equal to or less than the reference value (NO in step S134), the process proceeds to step S128.

[0082] As described above, the electronic device (e.g., information processing device 10) according to this embodiment includes display 14 having a plurality of pixels arranged thereon, camera 28, and a controller (e.g., host system 100). Light that has passed through a transparent area TA that is a partial area of ​​display 14 is incident on camera 28. The controller generates a difference image between a first image captured when display information is displayed in transparent area TA and a second image captured when display information is not displayed, and generates a restored image that is the difference between a third image captured when display information is displayed in transparent area TA and the difference image. According to this configuration, a difference image showing the influence of the display information is obtained as a difference image between a first image acquired when the display information is displayed in the transparent area TA and a second image acquired when the display information is not displayed in the transparent area TA. A restored image in which the influence of the display information has been removed from the third image is obtained by subtracting the difference image showing the influence of the display information from a third image captured when the display information is displayed in the transparent area TA. A restored image in which degradation of image quality caused by the display information is avoided or mitigated can be obtained while the display information is continued to be displayed on the display 14.

[0083] The controller may also obtain a first image and a second image from the camera 28 when the displayed information is still information. With this configuration, the first image and the second image are acquired at different times while the same still information is being displayed, so that the influence of the displayed information is accurately reflected in the differential image, and the influence of the displayed information in the third image can be more accurately removed.

[0084] Furthermore, the electronic device may include a detection unit (for example, a human sensor or an image recognition unit) that detects a person in the vicinity of the electronic device, and the controller may obtain the first image and the second image when no person is detected. According to this configuration, the first image and the second image are acquired when no person is detected around the electronic device. Therefore, since no person is included as a subject of the first image and the second image, the effect of the person's movement is not included in the difference image. Therefore, the effect of the display information in the difference image is accurately reflected, and the effect of the display information in the third image can be more accurately removed.

[0085] In addition, an influence map, which is an image captured by emitting light at a constant brightness for each light-emitting unit (e.g., pixel or sub-pixel) arranged in the transmissive region TA, may be set in advance, and the controller may accumulate the influence map based on a signal value representing the display information for each light-emitting unit to generate a difference image. According to this configuration, a difference image showing the influence of the display information is generated by accumulating the influence map for each pixel based on the signal value for each pixel of the display information displayed in the transparent area TA. Since the influence on the captured image according to the display information at each point in time is evaluated, even if the display information changes over time, the influence of the display information in the third image can be more accurately removed.

[0086] Furthermore, the controller may estimate, based on the imaging parameters of the third image, a difference image that is to be captured using the imaging parameters. According to this configuration, the influence of the display information obtained using the imaging parameters of the third image is estimated from the influence map obtained in advance. Therefore, even if the imaging parameters of the third image are different from the imaging parameters used to obtain the influence map, the influence of the display information in the third image can be more accurately removed.

[0087] Furthermore, an influence map may be set in advance for each imaging parameter, and the controller may specify an influence map corresponding to the imaging parameter of the third image, and generate a difference image using the specified influence map. According to this configuration, since the difference image is generated using the influence map corresponding to the imaging parameters of the third image, it is possible to more accurately remove the influence of the display information in the third image.

[0088] Furthermore, the controller may reduce the luminance of the display information when the component ratio of the difference image in the third image is greater than a predetermined reference value of the component ratio. According to this configuration, when the difference image component is relatively large in the third image, the luminance of the display information is reduced, so that the influence of the display information in the newly captured third image can be reduced and the image quality can be improved.

[0089] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above embodiment, and the present invention also includes designs that do not deviate from the gist of the present invention. The configurations described in the above embodiment can be combined in any combination. [Explanation of symbols]

[0090] 10...information processing device, 10a...first housing, 10b...second housing, 11...processor, 12...main memory, 13...video subsystem, 14...display, 14b...substrate, 14p...pixel, 21...chipset, 22...ROM, 23...auxiliary storage device, 24...audio system, 25...communication module, 26...input / output interface, 28...camera, 31...EC, 32...input device, 33...power supply circuit, 34...battery, 100...host system, 102...display control unit, 104...differential image generation unit, 106...image restoration unit, 108...output processing unit, 110...influence map setting unit, 120...storage unit, 121a, 121b...hinge mechanism, ax...rotation axis, NA...normal area, TA...transparent area, SA...screen area

Claims

1. The present invention includes a display having a plurality of pixels arranged thereon, a camera, and a controller, Light that has passed through a transmissive region that is a part of the display is incident on the camera, The controller: generating a difference image between a first image acquired when display information is displayed in the transparent area and a second image acquired when the display information is not displayed; A restored image is generated, which is a difference between a third image captured when display information is displayed in the transparent area and the difference image. electronic equipment.

2. The controller: When the display information is still information, the first image and the second image are acquired from the camera.

2. The electronic device according to claim 1.

3. A detection unit is provided for detecting a person in the vicinity of the device, When the person is not detected, The controller: Acquiring the first image and the second image 2. The electronic device according to claim 1.

4. An influence map is set in advance, which is an image captured by making each light-emitting unit arranged in the transmission region emit light at a constant luminance; The controller: The influence map is accumulated based on a signal value representing the display information for each of the light emission units to generate the difference image.

2. The electronic device according to claim 1.

5. The controller: The difference image captured using the imaging parameters of the third image is estimated based on the imaging parameters of the third image.

5. The electronic device according to claim 4.

6. The influence map is set in advance for each imaging parameter; The controller: Identifying the influence map corresponding to an imaging parameter of the third image; The difference image is generated using the identified influence map.

5. The electronic device according to claim 4.

7. The controller: When the component ratio of the difference image in the third image is greater than a predetermined reference value of the component ratio, Decreasing the brightness of the displayed information 2. The electronic device according to claim 1.

8. The present invention includes a display having a plurality of pixels arranged thereon, a camera, and a controller, An image processing method for an electronic device in which light that has passed through a transmissive region that is a part of the display is incident on the camera, comprising: The electronic device comprises: generating a difference image between a first image acquired when display information is displayed in the transparent area and a second image acquired when the display information is not displayed; A restored image is generated, which is a difference between a third image captured when display information is displayed in the transparent area and the difference image. Image processing methods.

Images