Electronic apparatus and image processing method

Abstract

To remove a virtual image while suppressing deterioration of image quality.SOLUTION: An electronic apparatus comprises: a display with a plurality of pixels aligned therein; a camera; and a controller. Light transmitted by a partial area of the display with pixels aligned therein is incident on the camera. The controller determines a high luminance area with a luminance exceeding a fixed value on the basis of the luminance distribution of an image captured by the camera, determines virtual image distribution on the basis of a virtual image pattern for a point light source and the high luminance area, estimates a corrected image on the basis of the captured image using a machine learning model obtained by learning the relation between a first image captured through the display and a second image captured without transmitted by the display, determines an application intensity of the corrected image on the basis of the virtual image distribution, and synthesizes the corrected image and the captured image on the basis of the application intensity.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present application relates to an electronic device and an image processing method, for example an electronic device having a display and a camera. [Background technology]

[0002] Some electronic devices, such as clamshell-type PCs (Personal Computers), are equipped with a display and a camera in their housings. In order to obstruct the screen display as little as possible, the camera is sometimes provided around the periphery of the screen area of ​​the display and supported by a bezel. Also, in electronic devices with a relatively small screen area, an imaging area may be provided around the screen area, and an imaging unit may be provided on the back side of the imaging area. The screen occupancy rate is limited by the camera being provided away from the screen area.

[0003] In recent years, in response to demands for devices with a high screen occupancy rate, electronic devices that adopt the CUD (Camera Under Display) method have been proposed. The CUD method is a configuration in which a camera is installed on the back side of an imaging area that forms part of the screen area. The camera captures an image that appears in light that has passed through the imaging area. For example, Patent Document 1 describes a display device that covers the device body, has an element area positioned behind the display area, and includes a photosensitive element in the element area that collects light that passes through the display area. The photosensitive element includes a camera.

[0004] In the CUD method, the pixels arranged in the imaging area are turned off when capturing an image. A significant portion of the incident light from the subject is reflected by the surface of the pixels arranged on the display panel and diffracted at the outer edge of the pixels. Reflection and diffraction can be the main cause of artifacts that tend to appear around the image of the light source. These artifacts cannot be eliminated by general image processing such as color conversion, noise reduction, super-resolution, sharpening, and gloss removal. Therefore, it is conceivable to eliminate the artifacts that appear in the captured image by using an AI (Artificial Intelligence) model that has learned the relationship between the target image captured without passing through the display and the captured image captured through the display. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2022-513506 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the estimation accuracy tends to decrease for estimated images estimated from captured images whose relationship to the target image is unknown. For example, in the estimated image, virtual images that cannot occur in the target image may appear, and the brightness distribution of the subject other than the light source may vary significantly. On the other hand, in commercial electronic devices with built-in displays, it is not realistic to detach the display to obtain a captured image and a target image corresponding to the captured image. In other words, since there are differences in the imaging conditions between the captured image and the target image other than the presence or absence of a display, these differences cannot be explained by the AI ​​model. For example, when the same field of view is captured using separate cameras, the brightness and color distribution of the captured image may vary depending on the viewpoint between the cameras and the properties of the components. [Means for solving the problem]

[0007] The present application has been made to solve the above-mentioned problems, and an electronic device according to one aspect of the present application includes a display on which a plurality of pixels are arranged, a camera, and a controller, wherein light that has passed through a portion of the display is incident on the camera, and the controller determines a high-brightness area where the brightness exceeds a certain value based on the brightness distribution of an image captured by the camera, determines a virtual image distribution based on a virtual image pattern for a point light source and the high-brightness area, estimates a correction image based on the captured image using a machine learning model that has learned the relationship between a first image captured through the display and a second image captured without passing through the display, determines an application intensity of the correction image based on the virtual image distribution, and combines the correction image and the captured image based on the application intensity.

[0008] In the electronic device described above, the controller may determine a specific area representing a specific subject from the captured image, and apply the captured image to the specific area without applying the correction image.

[0009] In the above electronic device, the controller may determine the virtual image pattern in accordance with an imaging parameter of the camera.

[0010] In the electronic device described above, the imaging parameters may include a parameter related to exposure of the camera.

[0011] In the electronic device described above, the camera may be installed behind a display surface of the display.

[0012] An image processing method according to a second aspect of the present application is an image processing method for an electronic device comprising a display on which a plurality of pixels are arranged and a camera, in which light that has passed through a portion of the display is incident on the camera, wherein the electronic device determines a high-brightness area whose brightness exceeds a certain value based on the brightness distribution of an captured image captured by the camera, determines a virtual image distribution based on the high-brightness area, a virtual image pattern for a point light source, estimates a correction image based on the captured image using a machine learning model that has learned the relationship between a first image captured through the display and a second image captured without passing through the display, determines an application intensity of the correction image based on the virtual image distribution, and combines the correction image and the captured image based on the application intensity. Effect of the Invention

[0013] According to this embodiment, it is possible to remove the virtual image while suppressing deterioration in image quality. [Brief description of the drawings]

[0014] [Figure 1] 1 is a front view showing an example of the external configuration of an information processing device according to an embodiment of the present invention; [Diagram 2] 1 is a cross-sectional view illustrating a cross section of an information processing device according to an embodiment of the present invention. [Diagram 3] 1 is a cross-sectional view showing an example of the configuration of a camera according to an embodiment of the present invention. [Figure 4] 1 is a schematic block diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment of the present invention. [Diagram 5] 1 is a schematic block diagram illustrating an example of a functional configuration of an information processing device according to an embodiment of the present invention. [Figure 6] FIG. 2 is an explanatory diagram illustrating a machine learning model according to the embodiment; [Figure 7] FIG. 2 is a diagram illustrating an example of an imaging environment for a first image and a second image. [Figure 8] 1 is a flowchart illustrating an image processing method according to the present embodiment. [Figure 9] 10 is a flowchart illustrating a weighting filter setting process according to the present embodiment. [Figure 10] FIG. 13 is a diagram showing an example of a virtual image pattern. [Figure 11] FIG. 13 is a diagram showing another example of the virtual image pattern. [Figure 12] FIG. 4 is a diagram showing an example of a captured image. [Figure 13] FIG. 13 is a diagram illustrating an example of a corrected image. [Figure 14] FIG. 4 is a diagram illustrating an example of a light source region. [Figure 15] FIG. 13 is a diagram showing an example of a comprehensive virtual image distribution. [Figure 16] FIG. 13 is a diagram illustrating an example of a virtual image distribution. [Figure 17] FIG. 13 is a diagram illustrating an example of a specific region. [Figure 18] FIG. 13 is a diagram illustrating an example of an output image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an electronic device according to an embodiment of the present application will be described with reference to the drawings. In the following description, an example will be taken of an information processing device 10 configured as a clamshell type PC as an electronic device according to the present embodiment. FIG. 1 is a front view showing an example of the external configuration of the information processing device 10 according to the present embodiment. FIG. 2 is a cross-sectional view showing an example of a cross section of the information processing device 10 according to the present embodiment.

[0016] The information processing device 10 includes a first housing 10a and a second housing 10b. The first housing 10a is rotatable relative to the second housing 10b, and the angle between the surface of the first housing 10a and the surface of the second housing 10b (sometimes referred to as the "opening / closing angle" in this application) is variable. The long side of the first housing 10a and the long side of the second housing 10b are connected using hinge mechanisms 121a and 121b. The hinge mechanisms 121a and 121b enable the first housing 10a to rotate relatively to the second housing 10b around a rotation axis ax. The direction of the rotation axis ax is parallel to both the long side of the first housing 10a and the long side of the second housing 10b. The hinge mechanisms 121a and 121b can maintain an arbitrary opening / closing angle θ even when some torque is applied.

[0017] The inside of the first housing 10a is formed in a box shape, and is equipped with a display 14 and a camera 28. Other members are stored inside the second housing 10b. In addition, a keyboard 107 and a touch pad 109 are mounted on the surface of the second housing 10b. When the first housing 10a is opened relative to the second housing 10b, a user facing the front of the first housing 10a can see the display information displayed on the display 14 and can perform input operations on the keyboard 107 and the touch pad 109. At this time, the camera 28 can capture an image of the user's head. In the following description, the first housing 10a and the second housing 10b may be collectively referred to simply as "housings".

[0018] The display 14 has a substantially flat plate shape and is configured as a display panel. The display 14 covers most of the surface of the first housing 10a, and its periphery is supported by the first housing 10a. The display 14 includes a substrate 14b. A plurality of pixels 14p are arranged two-dimensionally at regular intervals on the substrate 14b. The region in which the plurality of pixels 14p are arranged forms a screen region SA. Display information is expressed by the brightness or color distribution in the screen region SA. In the transmissive region TA, which is a part of the screen region SA, the pixels 14p are arranged more sparsely than in the normal region NA, which is the surrounding region. In the transmissive region TA, the adjacent pixels 14p are arranged with gaps between them without being in complete contact with each other. Light incident on the transmissive region TA passes through the gaps between the pixels and enters the optical system of the camera 28. The pixel interval in the transmissive region TA may be equal to the pixel interval in the normal region NA. In that case, the size of the pixels arranged in the transmissive region TA may be smaller than the size of the pixels arranged in the normal region NA.

[0019] Next, a configuration example of the camera 28 according to this embodiment will be described. Fig. 3 is a cross-sectional view showing a configuration example of the camera 28 according to this embodiment. The camera 28 has an optical system, and captures an image of a subject distributed within a field of view, which is represented by light incident on the optical system from the subject. The camera 28 includes an objective lens 28l, an aperture 28d, a shutter 28s, an optical filter 28of, an image sensor 28is, an autofocus mechanism 28af, and a camera controller 28ct.

[0020] The objective lens 28l receives incident light that has passed through the transmission area TA of the display 14. The objective lens 28l collects the incident light and presents it on the imaging surface of the imaging element 28is via the aperture 28d, the shutter 28s, and the optical filter 28of. The objective lens 28l is supported by an autofocus mechanism 28af, and is moved in the direction of its own optical axis under the control of the camera controller 28ct, making it possible to adjust the distance (focusing distance) from the objective lens 28l to the imaging element 28is (focusing control).

[0021] The aperture 28d has a circular opening in the center, and the incident light transmitted through the objective lens 28l passes through the opening. The aperture 28d varies the effective diameter of the opening by displacing the aperture blades around the opening under the control of the camera controller 28ct (aperture control). The shutter 28s is controlled to open and close under the control of the camera controller 28ct. The exposure time of the shutter 28s is controlled by the camera controller 28ct (exposure control). The optical filter 28of covers the surface of the image sensor 28is, and transmits a portion of the incident light that has passed through the objective lens 28l when the shutter 28s is open. The optical characteristics of the optical filter 28of are added to the transmitted light that has passed through the optical filter 28of. For example, the spectral characteristics (color) are set as the optical characteristics. The optical filter 28of transmits light of a color to be received by each sub-pixel as a main component. It is determined in advance for each sub-pixel constituting each pixel whether it is to receive red, green, or blue light.

[0022] The imaging element 28is has an imaging surface. A plurality of light receiving elements are two-dimensionally arranged on the imaging surface. Each light receiving element corresponds to a sub-pixel, detects light transmitted through the optical filter 28of, and generates a voltage according to the intensity of the detected light, i.e., brightness. A color is expressed by a set of brightness for each sub-pixel that constitutes one pixel. The camera controller 28ct generates image data showing the color distribution for each pixel, and outputs it to the host system 100 that constitutes the main body of the information processing device 10. The color distribution for each pixel represents a captured image. The camera controller 28ct can open the shutter 28s in response to an imaging command from the host system 100 to obtain a captured image.

[0023] In the present application, the camera 28 can use a known method for determining the imaging parameters. The process for determining the imaging parameters can include, for example, focus control, exposure control, and aperture control. The focus control determines the focal distance. The exposure control determines the exposure time. The aperture control determines the aperture value (F value). The aperture value is a parameter for brightness and depth of field. The aperture value corresponds to a quotient obtained by dividing the focal length of the objective lens 28l by the effective aperture of the aperture 28d. The imaging parameters depend on the environment in the field of view, such as the spatial distribution of the subject in the field of view, brightness, etc. In general, the brighter the environment, the shorter the exposure time and the smaller the aperture value. The more the subjects are close to the camera 28, the shorter the focus distance is adjusted so that the degree of focus (focus) for these subjects is higher. For example, the ratio of high spatial frequency components to low spatial frequency components in the captured image is used as the degree of focus. Moreover, the aperture value is adjusted to be larger as the variance of the distance from the camera 28 to the subject becomes greater. Since these imaging parameters are mutually dependent, they are determined so as to comprehensively optimize image quality. Image quality parameters may generally be expressed as a weighted sum of multiple factors such as focus and contrast. In this case, a weighting coefficient for each factor is set in advance as the contribution to that factor. The imaging parameters may be set in response to a user operation.

[0024] The information processing device 10 estimates a light source region representing a light source based on the luminance distribution of an image captured by the camera 28, and determines a virtual image distribution around the light source region based on a preset virtual image pattern. The information processing device 10 estimates a correction image based on the image captured by the camera 28, using a learning model that has previously learned the relationship between a non-transparent display image captured without passing through the display 14 and a transparent display image captured through the display 14. The information processing device 10 determines an application intensity of the correction image based on the estimated virtual image distribution, and combines the correction image with the captured image based on the determined application intensity.

[0025] Next, a hardware configuration example of the information processing device 10 according to the present embodiment will be described. Fig. 4 is a schematic block diagram showing a hardware configuration example of the information processing device 10 according to the present embodiment. The information processing device 10 includes a processor 11, a main memory 12, a video subsystem 13, a display 14, a chipset 21, a ROM 22, an auxiliary storage device 23, an audio system 24, a communication module 25, an input / output interface 26, a camera 28, an EC 31, an input device 32, a power supply circuit 33, and a battery 34.

[0026] The processor 11 is a core processing device that executes various arithmetic processing instructed by commands written in software (programs). The processing executed by the processor 11 includes reading and writing data from and to storage media such as the main memory 12 and the auxiliary storage device 23, and input and output with other devices. The processor 11 includes at least one CPU. The CPU controls the operation of the entire information processing device 10. The CPU executes processing based on programs such as an OS (Operating System), firmware, device drivers, utilities, and application programs (also referred to as "apps" in this application). Note that in this application, executing processing instructed by commands written in various programs may be referred to as "executing a program" or "running a program", etc.

[0027] The main memory 12 is a writable memory used as a read area for the execution program of the processor 11 or as a work area for writing processing data for the execution program. The main memory 12 is composed of, for example, a plurality of DRAM (Dynamic Random Access Memory) chips. The processor 11 and the main memory 12 correspond to the minimum hardware constituting the host system 100 (described later). The host system 100 is a computer system that forms the core of the information processing device 10.

[0028] Video subsystem 13 is a subsystem for realizing functions related to image display, and includes a video controller. The video controller processes drawing commands from processor 11, writes the obtained drawing information to a video memory, and also reads the drawing information from the video memory and outputs it to display 14 as display data indicating display information (image processing).

[0029] The display 14 displays a display screen based on the display data input from the video subsystem 13. The display 14 may be, for example, any of a liquid crystal display (LCD) and an organic light emitting diode (OLED) display.

[0030] The chipset 21 includes a plurality of controllers and can be connected to a plurality of devices so as to input and output various data. The controller is, for example, any one or a combination of bus controllers such as MIPI (Mobile Industry Processor Interface), USB (Universal Serial Bus), Serial ATA (AT Attachment), SPI (Serial Peripheral Interface) bus, PCI (Peripheral Component Interconnect) bus, PCI-Express bus, and LPC (Low Pin Count). The plurality of devices include, for example, a ROM 22, an auxiliary storage device 23, an audio system 24, a communication module 25, an input / output interface 26, a camera 28, and an EC 31, which will be described later.

[0031] A ROM (Read Only Memory) 22 mainly stores system firmware, firmware for controlling the operation of the EC 31 and other devices, etc. The ROM 22 may be, for example, any of an EEPROM (Electrically Erasable Programmable Read Only Memory) and a flash ROM.

[0032] The auxiliary storage device 23 stores various data used in the processing of the processor 11 and other devices, or various data acquired by the processing, various programs, etc. The auxiliary storage device 23 may be, for example, any one of an SSD (Solid State Drive), an HDD (Hard Disk Drive), etc., or any combination thereof.

[0033] The audio system 24 is connected to a microphone and a speaker (not shown) and records, plays, and outputs audio data. The microphone and the speaker may be built into the information processing device 10 or may be separate from the information processing device 10.

[0034] The communication module 25 connects to a communication network wirelessly or by wire. The communication module 25 communicates various data with other devices connected to the communication network. The communication module includes, for example, a wireless LAN (Local Area Network), and enables various data to be transmitted and received between devices according to a predetermined wireless communication method (for example, IEEE802.11). In the wireless LAN, communication between devices is performed via an access point.

[0035] The input / output interface 26 is connected to various devices such as peripherals in a wired or wireless manner. The input / output interface 26 is, for example, a connector for inputting and outputting data in a wired manner in accordance with the USB standard.

[0036] The camera 28 captures an image including an image of a subject located within the field of view. The camera 28 outputs image data indicating the captured image obtained by capturing the image to the processor 11 via the chipset 21.

[0037] The EC (Embedded Controller) 31 is a one-chip microcomputer that monitors and controls various devices (peripheral devices, sensors, etc.) regardless of the operating state of the system of the information processing device 10. The EC 31 includes a CPU, a ROM, a RAM, an A / D (Analog-to-Digital) input terminal for multiple channels, a D / A (Digital-to-Analog) output terminal, a timer, and digital input / output terminals (not shown) separate from the processor 11. For example, an input device 32, a power supply circuit 33, etc. are connected to the input / output terminals of the EC 31.

[0038] The input device 32 detects a user's operation and outputs an operation signal corresponding to the detected operation to the EC 31. The above-mentioned keyboard 107 and touch pad 109 correspond to the input device 32. The input device 32 may be a touch sensor, or may overlap the display 14 and be configured as a touch panel.

[0039] The power supply circuit 33 converts the voltage of DC power supplied from an external power supply or a battery 34 to a voltage required for the operation of each device constituting the information processing device 10, and supplies the power having the converted voltage to the device as a supply destination. The power supply circuit 33 executes power supply according to the control of the EC 31. The power supply circuit 33 includes a converter that converts the voltage of the power supplied to itself, and a power supply device that charges the battery 34 with the power whose voltage has been converted. The power supply device charges the battery 34 with the power that is supplied from the external power supply and that is not consumed by each device. When power is not supplied from the external power supply, or when the power supplied from the external power supply is insufficient, the power discharged from the battery 34 is supplied to each device as operating power. The battery 34 charges or discharges power using the power supply circuit 33. The battery 34 may be, for example, a lithium ion battery, a sodium ion battery, or the like.

[0040] Next, an example of the functional configuration of the information processing device 10 according to this embodiment will be described. Fig. 5 is a schematic block diagram showing an example of the functional configuration of the information processing device 10 according to this embodiment. The information processing device 10 includes a camera 28, a host system 100, a display 14, and a storage unit 120. Some or all of the functions of the host system 100 are realized by the processor 11 executing application programs (sometimes referred to as "apps" in this application) and other programs on the OS in cooperation with the main memory 12, the camera 28, the display 14, the storage unit 120, and other hardware.

[0041] The host system 100 includes an imaging control unit 102, a light source region estimation unit 104, a virtual image distribution estimation unit 106, a corrected image estimation unit 108, a specific region estimation unit 110, an application intensity setting unit 112, an image synthesis unit 114, and an output processing unit 116. The imaging control unit 102 generates an imaging command and outputs the generated imaging command to the camera 28. When an imaging command is input from the imaging control unit 102, the camera controller 28ct causes the imaging element 28is to capture an image of a subject located within the field of view.

[0042] Typical triggers for the imaging control unit 102 to issue an imaging command include, for example, when an operation signal generated in response to pressing a shutter button (not shown), when an operation signal indicating imaging is detected from the input / output interface 26, when the imaging control unit 102 is called by processing related to another application, and the like. The imaging control unit 102 may also notify the camera 28 of imaging parameters, such as one or any combination of focal distance, exposure time, F-number, etc., in association with the imaging command, and cause the camera 28 to capture an image according to the imaging parameters. The imaging control unit 102 may adopt imaging parameters instructed by an operation signal input from the input device 32 or the input / output interface 26, or may adopt imaging parameters instructed by execution of another application. In addition, when the camera controller 28ct controls imaging parameters in response to the environment within the field of view (automatic control), the imaging control unit 102 notifies the imaging control unit 102 of the imaging parameters determined by the imaging control unit 102 itself. The imaging control unit 102 may not notify the camera 28 of the imaging parameters set by the imaging control unit 102 itself.

[0043] The light source area estimation unit 104 estimates an area where a light source is represented as a subject as a light source area based on the luminance distribution of the captured image shown in the image data acquired from the camera 28. For example, the light source area estimation unit 104 refers to the image data, detects a pixel part where the luminance value is equal to or greater than a predetermined upper limit value as a bright area, and estimates an area where bright areas are adjacent to each other over a certain size or more as a light source area. When a color signal value is given for each pixel, the luminance value is given by a weighted average of the color signal value for each color. The light source area estimation unit 104 outputs light source area data indicating the estimated light source area to the virtual image distribution estimation unit 106.

[0044] The virtual image distribution estimation unit 106 estimates the virtual image distribution in the captured image based on the light source area data and the virtual image pattern input from the light source area estimation unit 104. The virtual image distribution estimation unit 106 estimates a comprehensive virtual image distribution by superimposing a virtual image pattern on each pixel included in the light source area indicated by the light source area data. The virtual image pattern corresponds to a virtual image distribution generated around a point light source. The virtual image pattern can also be regarded as a kind of point spread function. In the example of FIG. 10, the virtual image pattern is expressed using a pixel-by-pixel distribution value for each pixel in a two-dimensional space with the position of the point light source as the origin. The pixel-by-pixel distribution value is a value that expresses the brightness of each pixel, similar to the luminance value. In the virtual image pattern exemplified in FIG. 10, the virtual image becomes darker as it moves away from the point light source, and the brighter area spreads in the horizontal and vertical directions more than in the intersecting directions. The virtual image pattern may be actually measured in advance, or may be expressed using a geometric model.

[0045] When measuring the virtual image pattern, two types of captured images are captured by camera 28. One captured image is a light source image captured in a situation where a light source with a sufficiently small diameter and known brightness is placed within the field of view of camera 28. It is desirable that the diameter of the light source is equal to or smaller than the resolution of camera 28. In that case, the light source is regarded as a point light source. The other captured image is a background image captured in a situation where the light source is excluded from the field of view of camera 28. The difference value obtained by subtracting the signal value of the background image from the signal value of the light source image for each pixel corresponds to the virtual image pattern.

[0046] FIG. 11 illustrates a virtual image pattern expressed using a geometric model. In the illustrated geometric model, a virtual image is expressed that extends in the horizontal and vertical directions more than in the intersecting directions. The virtual image pattern is expressed using a radial component that depends on a radius with the position of the light source as the origin and a directional component that depends on the direction from the origin. The radial component is expressed using a function that attenuates to zero as the distance from the origin increases. The radial component is expressed using, for example, a Bessel function or a Hankel function. The directional component is expressed using a function that expresses the directional dependency of the spatial frequency up to at least four periods. The directional component is expressed using, for example, a sine function or a cosine function.

[0047] Returning to FIG. 5, the virtual image distribution estimation unit 106 calculates an accumulated value obtained by accumulating pixel-by-pixel distribution values ​​due to individual point light sources for each pixel as a virtual image distribution value. A comprehensive virtual image distribution is expressed by the virtual image distribution value for each pixel. The virtual image distribution estimation unit 106 can exclude the light source region from the comprehensive virtual image distribution and estimate the virtual image distribution occurring at the outer edge of the light source region. When excluding the light source region, the virtual image distribution estimation unit 106 sets the virtual image distribution value for each pixel belonging to the light source region among the virtual image distribution values ​​for each pixel that represent the comprehensive virtual image distribution to a predetermined reference value (for example, 0). The virtual image distribution estimation unit 106 outputs virtual image distribution data indicating the estimated virtual image distribution to the application intensity setting unit 112.

[0048] The corrected image estimation unit 108 estimates a corrected image based on a captured image shown in image data acquired from the camera 28 using a trained machine learning model. The machine learning model is also called an AI (Artificial Intelligence) model, and is a method of performing arithmetic processing on an input to derive an output, and is capable of acquiring a relationship between the input and the output by learning. In the machine learning model according to the present embodiment, a relationship is learned in advance in which an image captured through a display of another device (sometimes referred to as a "first image" in this application) is used as an input, and an image captured without transmitting the display (sometimes referred to as a "second image" in this application) is used as an output. By using a device having the same specifications as the information processing device 10 as the other device for learning, it is expected that a virtual image generated by the transmission of the display 14 from the captured image is removed in the corrected image. The corrected image estimation unit 108 outputs corrected image data indicating the estimated corrected image to the image synthesis unit 114.

[0049] The specific area estimation unit 110 estimates an area in which a specific subject is represented in the captured image shown in the image data acquired from the camera 28 as a specific area. The specific area estimation unit 110 can estimate the specific area by, for example, performing a known image recognition process on the captured image. As the specific subject, for example, a person (especially a head), a drawing, a flower, or other object in which the user is interested, or an object in which the visual effect of a virtual image is relatively significant, may be applied. The specific area can also be considered as a region of interest. The specific area estimation unit 110 outputs specific area data indicating the estimated specific area to the application intensity setting unit 112.

[0050] The application intensity setting unit 112 determines the application intensity of the correction image based on the virtual image distribution data input from the virtual image distribution estimation unit 106. The application intensity is a weight value indicating the degree of application to the correction image when the correction image and the captured image are combined. For example, the application intensity setting unit 112 determines the application intensity by normalizing the virtual image distribution value for each pixel indicated in the virtual image distribution data so that the maximum value is 1 and the minimum value is 0. The application intensity setting unit 112 configures a weight filter indicating the application intensity determined for each pixel. The application intensity setting unit 112 may set the specific region indicated in the specific region data input from the specific region estimation unit 110 as an application exclusion region that excludes application of the correction image. When setting the application exclusion region, the application intensity setting unit 112 sets the application intensity for each pixel in the specific region to 0, for example. The applied strength setting unit 112 outputs the constructed weighting filter to the image synthesis unit 114 .

[0051] The image synthesis unit 114 synthesizes the captured image indicated in the image data input from the camera 28 and the corrected image indicated in the corrected image data input from the corrected image estimation unit 108 into an output image using a weighting filter input from the application intensity setting unit 112. The image synthesis unit 114 determines the weighted sum of the color signal value of the corrected image for each pixel and the color signal value of the captured image as the color signal value of the output image (blending). When calculating the weighted sum, for example, the application intensity for that pixel is used as a weighting value for the color signal value of the corrected image, and a difference value obtained by subtracting the application intensity from 1 is used as a weighting value for the color signal value of the captured image (alpha blending). The components of the corrected image are predominant around the light source area, and the components of the captured image increase as the distance from the light source area increases. Therefore, it is possible to suppress the fluctuation in brightness and color tone that occurs in the corrected image while reducing the influence of the virtual image. The image synthesis unit 114 outputs output image data indicating the synthesized output image to the output processing unit 116. The method of synthesizing the output image from the corrected image and the captured image is not limited to this. For example, any method such as dither synthesis or multiplication may be used, or a combination of these methods may be used. Dither synthesis is a method of adopting a signal value of each pixel from one image with a probability corresponding to the applied intensity of the pixel, and adopting the signal value of the pixel of the other image when the signal value is not adopted. Multiplication is a method of normalizing the product of the signal value of each pixel of one image raised to the power of the applied intensity for that pixel and the signal value of the pixel of the other image raised to the power of the difference value obtained by subtracting the applied intensity from 1 for the signal value of that pixel, thereby determining the signal value for that pixel.

[0052] The output processing unit 116 performs processing related to the output of the output image data acquired from the image synthesis unit 114. The output processing unit 116 may output the output image data to another device via the display 14 or the input / output interface 26, or store the output image data in the storage unit 120. The output destination of the output image data may be instructed by an operation signal input from the input device 32 or the input / output interface 26, or may be instructed by the execution of another application.

[0053] Next, the machine learning model according to this embodiment will be described with reference to Fig. 6. The machine learning model according to this embodiment is used to estimate a corrected image from a captured image in the corrected image estimation unit 108 (inference). The machine learning model may be, for example, a neural network such as a convolutional neural network (CNN), or a mathematical model other than a neural network such as a random forest. To perform inference, the machine learning model is made to learn in advance the relationship between a first image (display-transparent image) that is an input and a second image (display-non-transparent image) that is an output.

[0054] Before learning, training data is prepared in advance. The training data includes a plurality of sets of a first image forming input data and a second image forming output data corresponding to the input data. In learning, a parameter set of the machine learning model is recursively updated so that the difference between the estimated data obtained by using the machine learning model for the input data and the output data corresponding to the input data is minimized as a whole of the training data. As the magnitude of the difference, for example, an index value such as a simple sum of squares, a weighted sum of squares, or a cross entropy of the color signal value for each pixel is used. In updating the parameter set, a method such as a steepest descent method or a stochastic gradient descent method is used. The information processing device 10 may or may not include a model learning unit (not shown) that learns the machine learning model. The corrected image estimation unit 108 may be set with parameters of the machine learning model obtained by learning from another device.

[0055] For each data set constituting the training data, it is preferable that the first image and the second image are captured under the same conditions as much as possible of a subject in the same field of view. In the example of FIG. 7, the first image is captured using the camera 28 provided in the information processing device 10. Incident light that has passed through the transparent area TA of the display 14 is incident on the optical system of the camera 28. The second image is captured using the camera 28b provided in front of the upper base of the display 14. The camera 28b is disposed in a position that does not cover the screen area of ​​the display 14 and is as close as possible to the transparent area TA. The optical axes of the cameras 28 and 28b are directed in the normal direction of the screen area of ​​the display 14. Under this arrangement, the fields of view of each of the cameras 28 and 28b include the head of a person located in front as a common subject and a light installed on the ceiling. Due to the difference in the positions of the cameras 28 and 28b, the fields of view of each captured image do not completely match. In a common region representing a common subject between the captured images, coordinate transformation may be performed so that an image of the common subject in one captured image matches the position and orientation of the image in the other captured image. When constructing training data, it is desirable to change at least one of the subject and its arrangement in the images for each data set.

[0056] Next, an example of an image processing method according to this embodiment will be described with reference to Fig. 8, which is a flowchart illustrating the image processing method according to this embodiment. (Step S102) The imaging control unit 102 causes the camera 28 to capture an image of a subject located within the field of view. (Step S104) The corrected image estimation unit 108 estimates a corrected image from the captured image captured by the camera 28 using a machine learning model.

[0057] (Step S106) The application strength setting unit 112 generates a weighting filter indicating the application strength of the corrected image for each pixel based on the virtual image distribution estimated from the captured image (weighting filter setting). (Step S108) The image synthesis unit 114 synthesizes the captured image and the corrected image using the generated weighting filter to generate an output image. (Step S110) The output processing unit 116 outputs the generated output image data to the display 14.

[0058] Next, an example of the weighting filter setting process according to this embodiment will be described below. Fig. 9 is a flowchart illustrating the weighting filter setting process according to this embodiment. (Step S106a) The light source region estimation unit 104 estimates, from the luminance distribution of the captured image captured by the camera 28, a region including pixels whose luminance values ​​are equal to or greater than an upper limit value, as the light source region. (Step S106b) The virtual image distribution estimation unit 106 estimates a comprehensive virtual image distribution by superimposing a virtual image pattern for each pixel arranged in the estimated light source area. The virtual image distribution estimation unit 106 excludes the light source area from the estimated comprehensive virtual image distribution and determines a virtual image distribution at the outer edge of the light source area.

[0059] (Step S106c) The specific region estimation unit 110 performs image recognition processing on the captured image captured by the camera 28, and estimates a region in which a specific subject is depicted as a specific region. (Step S106d) The application intensity setting unit 112 normalizes the virtual image distribution value forming the virtual image distribution to determine the application intensity. The application intensity setting unit 112 excludes the estimated specific region from the region where the application intensity is significantly greater than or equal to 0, and determines a weighting filter indicating the application intensity of the corrected image for each pixel. The determined weighting filter is used in the process of step S108.

[0060] Next, an example of a captured image will be described. Fig. 12 illustrates an example of a captured image captured by the camera 28 in the imaging environment illustrated in Fig. 7. The lower right part of the captured image shows an image of the upper body of a person sitting on a chair in a room. The background part of the captured image shows the walls and ceiling covering the room. The image of the lighting installed on the ceiling is shown as a haze. The haze occurring around the lighting corresponds to a virtual image that does not exist in reality. In particular, the haze of the two lighting fixtures in the upper central part is spatially connected.

[0061] FIG. 13 is a diagram illustrating a corrected image. The corrected image in FIG. 13 is an image estimated by using a machine learning model for the captured image in FIG. 12. According to the corrected image, the blurring around the illumination is suppressed more than in the captured image. However, the illumination of the input image is close to the camera 28, and there is a case where the image is captured in a form in which two illuminations are connected due to overexposure. This phenomenon may occur even when the image is captured with a camera that is not covered by the display, but it often occurs in the CUD method due to a decrease in the amount of light passing through the display 14. This phenomenon should be corrected in a natural way, but the brightness of a part of the bright area of ​​the captured image exceeds the upper limit and exceeds the range that can be corrected by the machine learning model. Therefore, it has been corrected in an unnatural way. In addition, the brightness distribution of the image of the person in the corrected image has changed from that of the captured image. More specifically, it is dark overall, and light and dark are emphasized. This change may change the impression on the user and may rather give a sense of incongruity to the corrected image. The facial expression of the person appearing in the corrected image is different from that appearing in the captured image. In contrast, in this embodiment, the components of the corrected image are increased around the light source and decreased in areas away from the light source, thereby mitigating this phenomenon.

[0062] Fig. 14 is a diagram illustrating a light source region. The light source region illustrated in Fig. 14 is a region estimated based on the luminance of the captured image in Fig. 12. The white parts represent the light source region, and the black parts represent regions that do not correspond to the light source region. The light source region estimation unit 104 detects bright parts that are continuous over a certain range as the light source region, thereby defining the outline of the light source region.

[0063] Fig. 15 is a diagram illustrating a comprehensive virtual image distribution. The comprehensive virtual image distribution illustrated in Fig. 15 is formed by superimposing the virtual image patterns illustrated in Fig. 10 for each pixel constituting the light source region illustrated in Fig. 14. Fig. 16 is a diagram illustrating an example of a virtual image distribution. The virtual image distribution illustrated in Fig. 16 is obtained by removing the light source region from the comprehensive virtual image distribution illustrated in Fig. 15. By removing the light source region from the comprehensive virtual image distribution, a virtual image distribution around the light source is extracted.

[0064] FIG. 17 is a diagram illustrating a specific region. The specific region is an area surrounded by a white dashed line. The illustrated specific region is an area where a person's head is likely to be present as a specific subject by performing image recognition processing on the captured image illustrated in FIG. 12. The application strength setting unit 112 determines a weighting filter so that the application strength of the correction image is higher in an area where the virtual image component is more prominent in the virtual image distribution illustrated in FIG. 17, and the application strength to the captured image is lower. However, the application strength setting unit 112 sets the application strength to 0 for the specific region so that the correction image is not applied.

[0065] FIG. 18 is a diagram illustrating an output image. The illustrated output image is an image obtained by synthesizing the captured image illustrated in FIG. 12 and the corrected image illustrated in FIG. 13 using a weighting filter in the image synthesis unit 114. The weighting filter is determined based on the virtual image distribution and the specific region illustrated in FIG. 17. The blurring around the light source is suppressed and the contour appears clearly. In addition, an unnatural luminance distribution due to the machine learning model does not occur in the region of the person's image. Therefore, the output image has both the suppression of the virtual image around the light source in the corrected image and the naturalness of the luminance distribution in the captured image, thereby obtaining a higher image quality than the corrected image.

[0066] In the above description, the virtual image pattern is mainly constant, but the present invention is not limited to this. The virtual image pattern may be variable depending on the imaging parameters of the camera 28. As described above, the imaging parameters are acquired by the imaging control unit 102. The imaging parameters may be determined depending on the environment in the field of view of the camera 28, or may be determined according to the user's operation. Even in this case, the user may determine the imaging parameters taking into account the environment in the field of view.

[0067] Therefore, virtual image pattern data indicating a relationship with a virtual image pattern for each imaging parameter is set in advance in the virtual image distribution estimation unit 106, and the set virtual image pattern data is referred to to determine virtual image pattern data corresponding to the imaging pattern acquired from the imaging control unit 102. The virtual image pattern data may have a format of a data table indicating a virtual image pattern for each imaging parameter, or may indicate a mathematical model in which the imaging parameters are input and the virtual image pattern is output. In this way, a virtual image distribution according to the imaging parameters is estimated.

[0068] The imaging parameters associated with the virtual image pattern may include at least a parameter related to the exposure of the camera 28. More specifically, the F-number or the effective aperture of the aperture 28d may be included. In general, the smaller the effective aperture of the aperture 28d, the more significant the diffraction of light incident on the optical system of the camera 28, and therefore the more likely the virtual image is to be prominent. Therefore, most of the variation in the virtual image pattern is explained by the change in the F-number or the effective aperture of the aperture 28d.

[0069] The virtual image pattern may also include distortion caused by the objective lens 28l that constitutes the optical system of the camera 28. Distortion is a phenomenon in which an image that appears at the periphery of the screen area is diffused toward the periphery rather than toward the center. Distortion is expressed for each pixel by the displacement of the virtual image pattern relative to that pixel. Therefore, virtual image pattern data indicating a relationship between a virtual image pattern for each set of an imaging parameter and a pixel (coordinate) may be set in advance in the virtual image distribution estimation unit 106. The virtual image distribution estimation unit 106 can refer to the set virtual image pattern data and determine virtual image pattern data corresponding to the imaging pattern acquired from the imaging control unit 102 for each pixel belonging to a bright part.

[0070] As described above, the present embodiment includes a display 14 in which a plurality of pixels are arranged, a camera 28, and a controller (for example, a host system 100). Light that has passed through a portion of the display 14 is incident on the camera 28. The controller determines a high-luminance region (for example, a light source region) in which the luminance exceeds a certain value based on the luminance distribution of the captured image captured by the camera 28, determines a virtual image distribution around the light source region based on a virtual image pattern for a point light source, estimates a correction image based on the captured image using a machine learning model that has learned the relationship between a first image captured through the display 14 and a second image captured without passing through the display 14, determines an application intensity of the correction image based on the virtual image distribution, and combines the correction image and the captured image based on the determined application intensity. The camera 28 may also be installed on the back of the display surface of the display 14. According to this configuration, a corrected image corresponding to a captured image is estimated using a machine learning model, a high-luminance area is determined based on the luminance distribution of the captured image, and the corrected image and the captured image are synthesized based on an adaptive strength determined based on a virtual image distribution determined from a virtual image pattern and the high-luminance area. The synthesized output image includes components of the corrected image with weights corresponding to the virtual image distribution around the light source area. A noticeable virtual image is suppressed near the light source area, and fluctuations in brightness and color in the corrected image in more distant areas are suppressed. Therefore, an output image with higher image quality than the corrected image can be obtained.

[0071] Furthermore, the controller may determine a specific area representing a specific subject from the captured image, and apply the captured image without applying the correction image to the determined specific area. According to this configuration, the image of the specific subject appears in the captured image without including components of the corrected image in which brightness or color has changed in the specific area, and the specific subject does not give the user an impression of something unusual.

[0072] The controller may also determine the virtual image pattern according to the imaging parameters of the camera 28. The imaging parameters may include parameters related to the exposure of the camera 28. According to this configuration, the virtual image distribution is determined based on a virtual image pattern that depends on the imaging parameters. Therefore, the virtual image distribution is more accurately estimated when the imaging parameters are adjusted according to the imaging environment within the field of view of the camera 28. By synthesizing the corrected image and the captured image using the applied intensity based on the estimated virtual image distribution, an output image with higher image quality can be obtained.

[0073] The imaging parameters may also include parameters related to the exposure of the camera 28. The virtual image pattern is adjusted based on a parameter related to the exposure of the camera 28, which is a main factor of the dependency of the virtual image pattern. Even if the number of parameters related to the adjustment of the virtual image pattern is reduced, the virtual image distribution can be estimated more accurately, so that the amount of processing required for adjusting the virtual image pattern can be reduced while suppressing deterioration in image quality.

[0074] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above embodiment, and the present invention also includes designs that do not deviate from the gist of the present invention. The configurations described in the above embodiment can be combined in any combination. [Explanation of symbols]

[0075] 10...information processing device, 10a...first housing, 10b...second housing, 11...processor, 12...main memory, 13...video subsystem, 14...display, 14b...substrate, 14p...pixel, 21...chipset, 22...ROM, 23...auxiliary storage device, 24...audio system, 25...communication module, 26...input / output interface, 28...camera, 28af...autofocus mechanism, 28ct...camera controller, 28d...aperture, 28is...imaging element, 28l...objective lens, 28of...optical filter , 28s...shutter, 31...EC, 32...input device, 33...power supply circuit, 34...battery, 100...host system, 102...imaging control unit, 104...light source area estimation unit, 106...virtual image distribution estimation unit, 107...keyboard, 108...corrected image estimation unit, 109...touchpad, 110...specific area estimation unit, 112...applied intensity setting unit, 114...image synthesis unit, 116...output processing unit, 120...storage unit, 121a, 121b...hinge mechanism, ax...rotation axis, NA...normal area, TA...transmission area, SA...screen area

Claims

1. The present invention includes a display having a plurality of pixels arranged thereon, a camera, and a controller, Light that has passed through a portion of the display is incident on the camera, The controller: A high brightness area having a brightness exceeding a certain value is determined based on a brightness distribution of the captured image captured by the camera; determining a virtual image distribution based on a virtual image pattern for a point light source and the high luminance area; estimating a corrected image based on the captured image using a machine learning model that has learned a relationship between a first image captured through the display and a second image captured without transmitting the display; determining an application intensity of the correction image based on the virtual image distribution; The corrected image and the captured image are combined based on the applied intensity. electronic equipment.

2. The controller: determining a specific region representing a specific subject from the captured image; Applying the captured image without applying the correction image to the specific region 2. The electronic device according to claim 1.

3. The controller: The virtual image pattern is determined according to an imaging parameter of the camera.

2. The electronic device according to claim 1.

4. The imaging parameters include a parameter related to the exposure of the camera.

4. The electronic device according to claim 3.

5. The camera is installed behind the display surface of the display. The electronic device according to any one of claims 1 to 3.

6. A display having a plurality of pixels arranged thereon and a camera, An image processing method for an electronic device in which light that has passed through a portion of the display is incident on the camera, comprising: The electronic device includes: A high brightness area having a brightness exceeding a certain value is determined based on a brightness distribution of the captured image captured by the camera; determining a virtual image distribution based on a virtual image pattern for a point light source and the high luminance area; estimating a corrected image based on the captured image using a machine learning model that has learned a relationship between a first image captured through the display and a second image captured without transmitting the display; determining an application strength of the correction image based on the virtual image distribution; The corrected image and the captured image are combined based on the applied intensity. Image processing methods.

Images