Image processing device, image processing method, and computer-readable storage medium

Abstract

The present invention provides an image processing device, an image processing method, and a computer-readable storage medium. The image processing device acquires first image data obtained by capturing an image of a subject, second image data obtained by capturing an image of the photographer and their surroundings, and third image data indicating the appearance of the imaging device that captured the image of the subject. The device reduces reflections in the first image data using a learned machine learning model that uses the first, second, and third image data as input data.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method and a computer-readable storage medium, and in particular to a technology for removing reflections in a captured image. Background Art

[0002] Images taken through glass and of shiny subjects may include reflections of the photographer or surrounding objects. Such reflections may not only blur the intended subject but also lead to the leakage of personal information.

[0003] In “A Computational Approach for Obstruction-Free Photography” by Tianfan Xue, Michael Rubinstein, Ce Liu, and William T. Freeman (ACM Transactions on Graphics, vol. 34, no. 4 (Proc. SIGGRAPH), August 2015), a technique is described for removing reflections in images taken through glass using visual disparity associated with background and foreground scenes.

[0004] However, the technology described in this patent document is limited in that it is necessary to capture images of a scene having both a background scene and a foreground scene from multiple viewpoints. Summary of the Invention

[0005] According to aspects of the present invention, limitations related to shooting scenes and imaging methods in an image processing apparatus and an image processing method capable of reducing reflections in a captured image are reduced.

[0006] According to one aspect of the present invention, there is provided an image processing device, comprising: one or more processors that execute a program stored in a memory and thereby serve as an acquisition unit, wherein the acquisition unit acquires first image data obtained by capturing an image of a subject, second image data obtained by capturing an image of a photographer and his or her surroundings, and third image data indicating the appearance of the camera device that captured the image of the subject; and an image processing circuit that uses a learned machine learning model that uses the first image data, the second image data, and the third image data as input data to acquire image data in which reflections in the first image data are reduced.

[0007] According to another aspect of the present invention, there is provided an image processing method performed by an image processing device, comprising: acquiring first image data obtained by capturing an image of a subject, second image data obtained by capturing an image of a photographer and his or her surroundings, and third image data indicating the appearance of the camera device that captured the image of the subject; and acquiring image data in which reflections in the first image data are reduced using a learned machine learning model that uses the first image data, the second image data, and the third image data as input data.

[0008] According to another aspect of the present invention, there is provided a computer-readable storage medium storing a program for causing a computer to execute the image processing method according to the present invention.

[0009] Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings). BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram illustrating an example configuration of a smartphone, which is an example of an image processing apparatus according to an embodiment of the present invention.

[0011] Figure 2A and 2B It shows Figure 1 FIG. 1 is a diagram showing an example of the appearance of a smartphone.

[0012] Figure 3 is a conceptual diagram of reflection reduction processing using a learned machine learning model according to this embodiment.

[0013] Figure 4A and 4C 4F to 4F are diagrams showing an example of a display of a reflection reduction imaging application according to an embodiment of the present invention, and Figure 4B is a diagram showing an example of the appearance of a subject.

[0014] Figure 5A and 5B : is a diagram showing an example of a display of a reflection-reduction imaging application according to an embodiment of the present invention.

[0015] Figure 6A and 6B is a flowchart related to the operation of the reflection reduction mode of the smartphone according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. While various features are described in the embodiments, the invention is not limited to requiring all of these features, and multiple features may be appropriately combined. Furthermore, in the accompanying drawings, the same reference numerals are assigned to the same or similar configurations, and redundant descriptions thereof are omitted.

[0017] Note that in the following embodiments, the present invention is described using an example of a smartphone equipped with a camera facing inward (photographer side) (inward-facing camera) and a camera facing outward (subject side) (outward-facing camera). However, the present invention can be implemented in other electronic devices (such as tablet terminals and laptop computers) equipped with inward-facing cameras and outward-facing cameras. In addition, the inward-facing camera and the outward-facing camera may be external devices that are attachable and detachable.

[0018] Furthermore, the present invention is not necessarily provided with an image capturing function, and the present invention may be implemented as any electronic device or system capable of acquiring, by any method, a captured image of a subject and image data of the photographer at the time of capturing the image. For example, the present invention may be implemented as an electronic device capable of acquiring such image data from a communicatively connected external device.

[0019] Note that ideally, the image of the subject and the image of the photographer at the time of capturing the image are captured at corresponding timings or simultaneously. However, as long as the time difference is short (e.g., within one second, within a few seconds, etc.), the capture timings may be different. Here, "simultaneously" may mean outputting a capture instruction to the image capture device to capture the respective images substantially simultaneously, or may mean overlapping of the image exposure periods.

[0020] Figure 1 1 is a diagram schematically showing an example configuration of a smartphone 100 , which is an example of an image processing apparatus according to the present invention, via functional blocks.

[0021] The internal bus 130 includes a data signal line, a control signal line, etc., and transmits data and control signals between connected functional blocks.

[0022] The CPU 101 is a main control unit and realizes various functions of the smartphone 100 by loading programs (OS, applications, etc.) stored in the nonvolatile memory 103 into the memory 102 and executing the programs to control the operations of the functional blocks.

[0023] The memory 102 is, for example, a RAM, and functions as a main storage device, a work memory, a buffer memory, a video memory, or the like.

[0024] The nonvolatile memory 103 is, for example, an electrically rewritable ROM. The nonvolatile memory 103 stores programs for execution by the CPU 101, various setting values, system data such as graphical user interface (GUI) data, user data such as still and moving image data captured by the user, applications, and the like.

[0025] The image processing circuit 104 is controlled by the CPU 101 to apply various types of image processing to image data obtained through imaging using the outward-facing camera 107, the standard inward-facing camera 109, and the wide-angle inward-facing camera 110, or to image data stored in the non-volatile memory 103. The image processing circuit 104 may be, for example, dedicated hardware such as an application-specific integrated circuit (ASIC) designed to implement specific functions. Alternatively, the image processing circuit 104 may be configured to implement specific functions by executing a program stored in the non-volatile memory 103 via a programmable processor such as a digital signal processor (DSP). Furthermore, the image processing circuit 104 may include a dedicated circuit for performing the high-speed calculations required for machine learning, such as deep learning. Furthermore, depending on the process, image processing may be applied by executing the program via the CPU 101 instead of the image processing circuit 104.

[0026] Here, the image processing applied by the signal processing circuit 104 includes pre-processing, color interpolation processing, correction processing, detection processing, data modification processing, evaluation value calculation processing, special effect processing, and the like.

[0027] Preprocessing includes noise reduction, defective pixel correction, A / D conversion, D / A conversion, etc.

[0028] The color interpolation process is a process for interpolating the values ​​of color components that are not obtained when an image is captured, and is also called a demosaic process or a synchronization process.

[0029] The correction processing includes white balance adjustment, grayscale correction (gamma processing), processing for correcting the influence of optical aberration or vignetting of the optical system, processing for correcting color, and the like.

[0030] The detection processing includes processing for detecting a feature region (for example, a face region or a human body region) or movement thereof, processing for recognizing a person, and the like.

[0031] The data modification process includes combining process, scaling process, encoding and decoding process, header information generation process, etc.

[0032] The evaluation value calculation process includes a process for generating a signal or an evaluation value used in automatic focus detection (AF), a process for calculating an evaluation value used in automatic exposure control (AE), and the like.

[0033] Special effect processing includes processing for adding blur, changing color tone, relighting processing, etc. Note that these processes are examples of image processing that can be applied by the image processing circuit 104 and are not intended to limit the image processing applied by the image processing circuit 104.

[0034] The display 105 is, for example, a touch liquid crystal display and is controlled by the CPU 101 to display images of a GUI of an OS or application (icons, windows, software keys, etc.), images captured by a camera, and the like.

[0035] The CPU 101 generates display data and stores the display data in a video storage area of ​​the memory 102. The display 105 generates a video signal based on the display data and displays the video signal on a display panel. Note that the video signal can also be supplied to an external display.

[0036] The operation unit 106 is a general name for a user-operable device such as a button, a switch (hardware or software), a touch panel, or the like. The CPU 101 monitors the operation unit 106 and performs an operation based on the operation target and operation content. Note that regarding touch operations on the display 105, the CPU 101 detects the touch position and recognizes a predetermined touch operation (tap, drag, pinch, wipe, etc.). The CPU 101 detects operations on input devices such as an externally connected keyboard or mouse.

[0037] Smartphone 100 includes one outward-facing camera 107 and two inward-facing cameras 109 and 110. The two inward-facing cameras 109 and 110 include single-focus lenses with different angles of view, and the lens of inward-facing camera 110 has a wider angle of view than the lens of inward-facing camera 109. Therefore, in this specification, these are referred to as the standard inward-facing camera 109 and the wide-angle inward-facing camera 110. However, "standard" and "wide-angle" refer to the relationship between the angles of view of the two inward-facing cameras and are not intended to indicate a specific angle of view.

[0038] The outward-facing camera 107 is a camera unit that includes, for example, an image sensor such as a CCD / CMOS image sensor, an imaging optical system (lens), a shutter, an aperture, a focus detection circuit, an A / D converter, and the like. The lens of the outward-facing camera 107 may be a zoom lens or a single-focus lens. Furthermore, the lens includes a focus lens. The outward-facing camera 107 processes both still and moving images. Image data obtained through imaging is transmitted to the image processing circuit 104, stored in a data file in a predetermined format, and stored in the non-volatile memory 103. If the display surface of the display 105 is defined as the front surface of the smartphone 100, the outward-facing camera 107 is provided on the rear surface of the smartphone 100.

[0039] The lighting device 108 is, for example, an auxiliary light source comprising one or more light-emitting elements (eg, LEDs). This is used for flash and video light when capturing images. The color of the light from the lighting device 108 may be adjustable.

[0040] The standard inward-facing camera 109 and the wide-angle inward-facing camera 110 are camera units having a configuration similar to that of the outward-facing camera 107, which includes a single-focus lens. The standard inward-facing camera 109 and the wide-angle inward-facing camera 110 are provided on the front surface of the smartphone 100. Therefore, the imaging directions of the outward-facing camera 107 and the two inward-facing cameras 109 and 110 differ by 180° from each other.

[0041] The communication I / F 111 is an interface for communicating with an external device that is communicatively connected via a wired or wireless connection. The communication I / F 111 can support multiple communication standards. In addition, the external device can be directly connected to the communication I / F 111 or can be connected via the network 112.

[0042] Figure 2A and 2B is a diagram showing an example of the external appearance of the smartphone 100 . Figure 2A is a plan view of a surface (hereinafter referred to as a front surface or a display surface) on which the display screen of the display 105 is located. Figure 2B is a plan view of a surface (back surface) parallel to the front surface. Figure 2A and 2B middle, Figure 1 The components shown are given Figure 1 The same reference numerals are used in the drawings.

[0043] The smartphone 100 of this embodiment has a reflection reduction mode that can be selected when capturing images using the outward-facing camera 107. When the reflection reduction mode is selected, the image capture using the outward-facing camera 107 and the image capture using one of the inward-facing cameras (e.g., the wide-angle inward-facing camera 110) are performed in parallel. Furthermore, the image of the photographer captured by the wide-angle inward-facing camera 110 is used to reduce reflections in the image of the subject captured by the outward-facing camera 107.

[0044] The reflection reduction mode can be selected by the user by operating the GUI on the screen of an application that can capture images using the outward-facing camera 107 and the inward-facing camera 110. Alternatively, the reflection reduction mode can be selected by default when a specific application is launched. In addition, the reflection reduction mode can be configured to be automatically selected when the image processing circuit 104 analyzes a moving image captured during the image capture standby state for EVF display and detects reflections.

[0045] Hereinafter, an example in which a reflection reduction process is performed by an application for capturing an image in a reflection reduction mode will be described. However, the reflection reduction process may be implemented as a single function of a general application or may be a function provided by an OS.

[0046] The reflection reduction processing of the image captured by the outward-facing camera 107 uses learned machine learning (ML) implemented using a neural network, for example. Figure 3 is a conceptual diagram of a reflection reduction process using a learned ML model 302 .

[0047] The machine learning model 302 outputs an image of a subject with reduced reflections, where the input data is:

[0048] 1) first image data (subject image) obtained by capturing an image of a subject by the outward-facing camera 107,

[0049] 2) second image data (image of the photographer) obtained by capturing an image of the photographer (photographing a subject) and the surrounding environment by the wide-angle inward-facing camera 110, and

[0050] 3) Third image data (device image) representing the appearance of an image pickup device (in this example, the smartphone 100 ) that captures an image of a subject.

[0051] Note that the image of the appearance of the smartphone 100 is, for example, an image of the back surface of the smartphone 100 which may be stored in advance in the nonvolatile memory 103 .

[0052] Note that the machine learning model 302 may be stored in the nonvolatile memory 103, or may be included in an external device capable of communicating with the smartphone 100. In the case where the machine learning model 302 is provided in the external device, the machine learning model 302 can be used by downloading it to the smartphone 100 via the communication I / F 111. In addition, instead of downloading the machine learning model 302, the input data 301 may be sent to the external device, and the calculation of the machine learning model 302 may be performed by the external device. In the case where the calculation of the machine learning model 302 is performed by the external device, the smartphone 100 receives the calculation result (the subject image with reduced reflection) from the external device.

[0053] In this example, the learning model 302 is either stored in the non-volatile memory 103 or downloaded to the smartphone 100 from an external device.

[0054] Note that the method for implementing the machine learning model 302 (the configuration and number of layers of the neural network, activation function, etc.) and the learning method (loss function, optimization algorithm, etc.) can be known technologies. In addition, the present invention does not depend on the method for implementing the machine learning model 302 and the learning method. Therefore, in this article, only a brief description of the learning method of the machine learning model 302 is given, and specific details will be omitted.

[0055] The machine learning model 302 can learn using supervised learning. Specifically, as input data, an image of a subject with reflection captured by the outward-facing camera 107, an image of the photographer captured by the wide-angle inward-facing camera 110, and an image of the device of the smartphone 100 are used. In addition, as data indicating a target result (i.e., target data), an image without reflection or with reduced reflection corresponding to the input data of the subject image is used in the training data. The target data can be generated by applying image processing to the subject image with reflection, or an image of the subject without reflection captured from the same viewpoint as the subject image with reflection can be used. For example, in an image captured through glass, an image with reflection can be acquired via normal photography, and then an image without reflection can be acquired by capturing the image after blocking the light around the smartphone 100.

[0056] The learned machine learning model 302 can be obtained by preparing various subjects and imaging environments as input data and learning data and training the machine learning model.

[0057] Next, we will describe the reasoning behind sending the wide-angle, inward-facing camera image to the machine learning model. Reflections in the subject image correspond to mirror images of images taken in the direction from the subject toward the outward-facing camera 107. Images taken in the direction from the subject toward the outward-facing camera 107 are significantly similar to images taken by the inward-facing camera. Based on the similarity between the subject image taken by the outward-facing camera 107 and the image of the photographer and surroundings taken by the inward-facing camera, the machine learning model can be trained to converge, thereby approximating the output of a result in which reflections are removed or reduced, i.e., target data.

[0058] In addition, images obtained by a wide-angle inward-facing camera 110, which is capable of capturing images of the photographer's surroundings over a wider range than the standard inward-facing camera 109, are more useful in detecting similarities in images with reflections than images obtained by a standard inward-facing camera 109.

[0059] Furthermore, since reflections of the smartphone 100 capturing the subject image may be included in the subject image, the exterior image of the smartphone 100 is sent to the machine learning model. Using the exterior image of the smartphone 100 as input data allows the machine learning model to easily detect reflections of the smartphone 100 in the subject image, thereby facilitating convergence training and ensuring that the output results approach the target data. This helps to achieve convergence in training and ensure that the output results approach the target data.

[0060] Figures 4A to 4F as well as Figure 5A and 5B 1 is a diagram showing an example of a display of a reflection reduction application. The CPU 101 executes the application, and the image processing circuit 104 and the like are controlled to display these displays.

[0061] Figure 4A The display state when the application is started is shown. A GUI for a camera button 401 is displayed at the bottom of the application screen. When the application is started, the camera standby state is activated, and video 402 captured by outward-facing camera 107 is displayed in real time on the application screen. CPU 101 continuously executes moving image capture processing by outward-facing camera 107, moving image data generation processing for display by image processing circuit 104, and moving image data display processing by display 105.

[0062] The user can check the imaging area, exposure status, etc. via the video 402. Touching the imaging button 401 captures an image. When a touch operation on the imaging button 401 is detected, the CPU 101 executes imaging processing in the reflection reduction mode.

[0063] Figure 4A A scene is shown that includes a television as the main subject and there are reflections of the user and their surroundings on the television screen. Figure 4B This image shows reflections removed through reflection reduction processing. By reducing reflections in this way, an image with the main subject standing out is obtained. Furthermore, when uploading images to the internet, the user's personal information can be protected from being leaked.

[0064] In the reflection reduction mode, the CPU 101 also captures a video with the wide-angle inward-facing camera 110 during standby. The CPU 101 then instructs the image processing circuit 104 to detect the presence of a user's reflection based on the image obtained by the inward-facing camera 110 and the image obtained by the outward-facing camera 107. The image processing circuit 104 detects the presence of a user's reflection using a known technique such as pattern matching and notifies the CPU 101 of the result.

[0065] When the image processing circuit 104 notifies the CPU 101 that there is a reflection of the user, the CPU 101 instructs the image processing circuit 104 to display the video obtained by the wide-angle inward-facing camera 110 in a picture-in-picture mode, for example. In response to this instruction, the image processing circuit 104 starts the picture-in-picture display of the video obtained by the wide-angle inward-facing camera 110 ( Figure 4C ).

[0066] Furthermore, the image processing circuit 104 superimposes an indicator on the video captured by the camera 107 facing outward to notify the user of the reflection. Figure 4D An example of a frame-shaped indicator superimposed in an area where a reflection of the user's face is present is shown. The image processing circuit 104 then detects an area in the video captured by the outward-facing camera 107 that is similar to the facial area detected in the video captured by the wide-angle inward-facing camera 110, and displays a superimposed frame image around that area.

[0067] Furthermore, on the video captured by the wide-angle inward-facing camera 110, the image processing circuit 104 displays an indicator indicating an area similar to the reflection of the face in the video captured by the outward-facing camera 107 ( Figure 4E ).

[0068] In this way, by using the face shown in the image captured by the wide-angle inward-facing camera 110 , the user can be informed that there is a high possibility that the reflection of the face in the image captured by the outward-facing camera 107 can be reduced.

[0069] However, there is a case where a reflection of a face is present in the image captured by outward-facing camera 107, but image processing circuit 104 determines that no reflection of a person's face appears in the image captured by wide-angle inward-facing camera 110. In this case, image processing circuit 104 displays an indicator on display 105 prompting the user to capture an image of the user's face using wide-angle inward-facing camera 110.

[0070] like Figure 4FAs shown, image processing circuit 104 displays a face-like frame and graphic as an indicator superimposed on the image captured by wide-angle, inward-facing camera 110. Note that based on the reflected image, the preferred size and position of the user's face within the imaging area of ​​wide-angle, inward-facing camera 110 can be determined, and the indicator can be displayed at that position and size based on the determination. Note that the indicator can be a message using words to convey detailed instructions. By displaying the indicator in this manner, the likelihood of more appropriately reducing reflections of the face in images captured by outward-facing camera 107 can be increased.

[0071] Note that regardless of how the indicator is displayed, the capture button 401 can be operated in a state where no face is shown in the wide-angle inward-facing camera 110. In this case, the CPU 101 instructs the image processing circuit 104 to display a message on the display 105 notifying the user that facial reflection cannot be reduced. Figure 5A is an example of a message.

[0072] exist Figure 5A In the state shown, for example, when a tap operation on the display 105 is detected, the CPU 101 instructs the image processing circuit 104 to display a dialog box on the display 105 for the user to confirm whether they wish to continue imaging. Figure 5B is an example of a confirmation dialog box.

[0073] The confirmation dialog box includes "Yes" and "No" buttons for indicating whether to continue shooting. When the tapping operation of the "Yes" button is detected, the CPU 101 starts the still image shooting operation. When the tapping operation of the "No" button is detected, the CPU 101 notifies the image processing circuit 104 that the confirmation dialog box is canceled. In this case, the display of the reflection reduction application returns to Figure 4F status.

[0074] Next, we will use Figure 6A and 6B The operation of the CPU 101 related to imaging in the reflection reduction mode is described in the flowchart shown in FIG. Figures 4A to 4F The camera environment shown in .

[0075] In step S601, the CPU 101 determines whether a user operation to turn on the reflection-reducing camera mode has been detected. Here, the operation of launching the reflection-reducing camera application installed in the smartphone 100 is considered to be the operation of turning on the reflection-reducing camera mode. Note that, instead of detecting the operation, it is possible to determine whether the current camera mode is set to the reflection-reducing camera mode. If the CPU 101 determines that a user operation to turn on the reflection-reducing camera mode has been detected, step S602 is executed. If this is not determined, step S601 is repeated.

[0076] In step S602, the CPU 101 starts the camera standby state operation. The CPU 101 starts shooting video using the camera 107 facing outward. In addition, the CPU 101 instructs the image processing circuit 104 to generate a video for display and displays the generated video. In this way, the video ( Figure 4A ).

[0077] Furthermore, the CPU 101 instructs the image processing circuit 104 to perform face detection processing on the video captured by the outward-facing camera 107. The image processing circuit 104 begins performing face detection processing on the frame images of the video and notifies the CPU 101 of the detection results (the number, size, and position of the detected facial regions, etc.). Note that the frequency of performing face detection processing can be set according to the frame rate or performance of the image processing circuit 104. The image processing circuit 104 can perform face detection processing continuously or can perform face detection processing on one frame each time it receives an instruction from the CPU 101. Thereafter, the CPU 101 executes step S603.

[0078] In step S603, the CPU 101 determines whether the angle of view of the inward-facing camera is set to wide angle. For example, the CPU 101 reads the current inward-facing camera angle of view setting from the memory 102 and determines whether the setting for using the wide-angle inward-facing camera 110 is enabled. Alternatively, the CPU 101 may determine whether the wide-angle inward-facing camera 110 is enabled from among the standard inward-facing camera 109 and the wide-angle inward-facing camera 110. If the angle of view of the inward-facing camera is determined to be wide angle, the CPU 101 executes step S605. If this is not determined, the CPU 101 executes step S604.

[0079] In step S604, the CPU 101 changes the angle of view setting of the inward-facing camera stored in the memory 102 to the wide angle. The CPU 101 then executes step S605.

[0080] In step S605, the CPU 101 analyzes the video captured by the outward-facing camera 107 starting in step S602. For example, the CPU 101 instructs the image processing circuit 104 to perform face detection processing on the video captured by the outward-facing camera 107. Note that if the image processing circuit 104 is instructed to continuously perform face detection processing in step S602, step S605 can be omitted. The CPU 101 then executes step S606.

[0081] In step S606, the CPU 101 determines whether there is a reflection of a face in the video captured by the outward-facing camera 107. For example, if the result of face detection processing performed by the image processing circuit 104 is obtained and a facial area is detected, the CPU 101 determines that there is a reflection of the face and executes step S607. If no facial area is detected, the CPU 101 determines that there is no reflection of the face and executes step S620.

[0082] In step S607, the CPU 101 starts capturing video using the wide-angle inward-facing camera 110 (this may have been started at step S605). Furthermore, the CPU 101 instructs the image processing circuit 104 to generate a video for display based on the video captured by the wide-angle inward-facing camera 110, and to display the generated video via a picture-in-picture display. In this way, the video showing the imaging area of ​​the outward-facing camera 107 and the video captured by the wide-angle inward-facing camera 110 are continuously displayed on the display 105 ( Figure 4C ). Then, the CPU 101 executes step S608.

[0083] In step S608, the CPU 101 instructs the image processing circuit 104 to display an indicator indicating the detected face area superimposed on the video captured by the camera 107 facing outward. Figure 4D As shown, an indicator indicating the detected facial region is superimposed and displayed on the video captured by the outward-facing camera 107. Note that the superimposed display of the indicator indicating the facial region may be displayed at the same time as the instruction to detect the facial region is sent. The CPU 101 then executes step S609.

[0084] In step S609, the CPU 101 instructs the image processing circuit 104 to apply face detection processing to the video captured by the wide-angle inward-facing camera 110. The CPU 101 then executes step S610. Note that the processing of step S609 may be included in the processing of step S607.

[0085] In step S610, the CPU 101 determines whether a face is included in the video captured by the wide-angle inward-facing camera 110 based on the result of the face detection processing received from the image processing circuit 104. If a face is detected in the video captured by the wide-angle inward-facing camera 110, the CPU 101 executes step S611. If a face is not detected, the CPU 101 executes step S614.

[0086] In step S611, the CPU 101 instructs the image processing circuit 104 to calculate a degree of agreement between the face detected in the video captured by the outward-facing camera 107 and the face detected in the video captured by the wide-angle, inward-facing camera 110. This degree of agreement indicates the degree of agreement between the two faces. Based on the facial regions detected in the two videos, the image processing circuit 104 converts the position, shape, and size of the eyes, nose, mouth, and other facial features, the overall shape of the face, protrusions and depressions in the face, and other facial features into numerical values. Furthermore, the image processing circuit 104 compares the numerical values ​​associated with the facial features and calculates a degree of agreement between the faces. For example, the degree of agreement between the faces can be the sum of the differences in numerical values, in which case a smaller value indicates a higher degree of agreement between the faces. The image processing circuit 104 notifies the CPU 101 of the calculated degree of agreement between the faces. Upon receiving the degree of agreement between the faces, the CPU 101 proceeds to step S612.

[0087] In step S612, the CPU 101 determines whether the face detected in the video captured by the outward-facing camera 107 and the face detected in the video captured by the wide-angle, inward-facing camera 110 are the faces of the same person based on the degree of consistency between the faces. For example, if the degree of consistency between the faces is within a threshold, the CPU 101 determines that the faces are of the same person and executes step S613. If the degree of consistency is equal to or greater than the threshold, the CPU 101 determines that the faces are of different people and executes step S615.

[0088] Note that the processing of steps S611 and S612 can be implemented through extrapolation processing using a learned model. In this case, image data of a face detected in a video captured by the outward-facing camera 107 and image data of a face detected in a video captured by the wide-angle, inward-facing camera 110 are used as input data for the learned model. The learned model then outputs a degree of consistency as output data. Furthermore, such a learned model can be generated, for example, through machine learning using a neural network. In this case, the machine learning can be supervised learning, which uses image data of a facial portion previously captured by the outward-facing camera 107 and image data of a facial portion previously captured by the wide-angle, inward-facing camera 110 as input data, and uses a degree of consistency between the two, obtained using a predetermined method, as target data.

[0089] In step S613, the CPU 101 instructs the image processing circuit 104 to display an indicator indicating the detected face detection superimposed on the video captured by the wide-angle inward-facing camera 110. Figure 4E As shown, an indicator indicating the detected facial region is superimposed and displayed on the video captured by the wide-angle inward-facing camera 110. Note that the superimposed display of the indicator indicating the facial region may be displayed at the same time as the instruction to detect the facial region is sent. The CPU 101 then executes step S620.

[0090] In step S614, the CPU 101 instructs the image processing circuit 104 to display an indicator (face position guide) for prompting the user to change his position so that the user's face can be captured by the wide-angle inward-facing camera 110. For example, in response to this instruction, the image processing circuit 104 displays a frame and a graphic (such as a face) superimposed in the central area of ​​the video captured by the wide-angle inward-facing camera 110. Figure 4F ), and the display is Figure 5A Then, the CPU 101 executes step S615.

[0091] In step S615, the CPU 101 determines whether the user has performed a camera operation. Specifically, the CPU 101 determines whether the camera button 401 has been touched. If the CPU 101 determines that the camera operation has been performed, the CPU 101 executes step S616. If it has not been determined that this is the case, the CPU 101 executes step S605.

[0092] In step S616, the CPU 101 instructs the image processing circuit 104 to display a message in a superimposed manner indicating that the facial reflection cannot be reduced. Here, the image processing circuit 104 displays a message on the display 105 indicating that "the reflection of the cameraman may be in the shooting. Since the cameraman cannot be recognized with the camera facing inward, the reflection may not be reduced" ( Figure 5A ). Then, the CPU 101 executes step S617.

[0093] In step S617, the CPU 101 determines whether a confirmation operation in response to the message has been detected, for example, a tap operation on the display 105. In the case where the CPU 101 determines that a confirmation operation has been performed, the CPU 101 instructs the image processing circuit 104 to display a dialog box on the display 105 for the user to confirm whether they wish to continue imaging. Here, the dialog box including a message indicating "Do you wish to continue?" and "Yes" and "No" buttons is displayed on the display 105 by the image processing circuit 104 ( Figure 5B ). Note that, instead of detecting the confirmation operation, it may be determined whether a predetermined amount of time has elapsed. The CPU 101 then executes step S618.

[0094] In step S618, the CPU 101 determines whether a stop-recording operation has been detected. Specifically, the CPU 101 determines whether an operation of pressing the "No" button in the confirmation dialog box displayed in step S617 has been detected. If the CPU 101 determines that a stop-recording operation has been performed, the CPU 101 executes step S605. If it has not been determined that this is the case, the CPU 101 executes step S619.

[0095] In step S619, the CPU 101 determines whether a continue shooting operation has been detected. Specifically, the CPU 101 determines whether an operation of pressing the "Yes" button in the confirmation message displayed in step S617 has been detected. If the CPU 101 determines that a continue shooting operation has been performed, the CPU 101 executes step S621. If it has not been determined that this has been performed, the CPU 101 executes step S618.

[0096] In step S620, as in step S615, the CPU 101 determines whether the user has performed an image pickup operation. If the CPU 101 determines that an image pickup operation has been performed, the CPU 101 executes step S621. If it has not determined that this is the case, the CPU 101 executes step S605.

[0097] In step S621, the CPU 101 performs still image capture from the outward-facing camera 107 and the wide-angle inward-facing camera 110. The aperture, shutter speed, and other parameters used for capturing the still image are set by the CPU 101 based on the evaluation value generated by the image processing circuit 104 based on the video captured by the outward-facing camera 107. In addition, the CPU 101 performs automatic focus detection of the optical system based on the evaluation value generated by the image processing circuit 104 based on the video captured by the outward-facing camera 107.

[0098] Furthermore, the CPU 101 instructs the image processing circuit 104 to generate still image data for recording based on the still image data captured by the outward-facing camera 107, and to generate still image data for use in the reflection reduction mode based on the still image data captured by the wide-angle, inward-facing camera 110. The image processing circuit 104 applies image processing to the still image data that prioritizes image quality over image processing for generating video data for display. The CPU 101 then executes step S622.

[0099] In step S622, the CPU 101 instructs the image processing circuit 104 to apply reflection reduction processing to the still image data captured by the outward-facing camera 107 in step S621. The image processing circuit 104 reads the learned machine learning model 302 and the appearance image of the smartphone 100 from the non-volatile memory 103 and loads them onto the internal memory or storage 102. The image processing circuit 104 then sends the still image data for recording, the still image data generated from the still image data captured by the wide-angle inward-facing camera 110, and the appearance image of the smartphone 100 to the learned machine learning model 302 as input data. The image processing circuit 104 stores the image with reduced reflection obtained from the calculation of the learned machine learning model 302 as output in the non-volatile memory 103.

[0100] In this manner, the image processing device according to this embodiment can reduce reflections in an image of a subject by sending an image of the subject, an image of the camera person and the surroundings, and an image of the appearance of the camera device as input data to a learned machine learning system. By using the image of the camera person and the surroundings, and the image of the appearance of the camera device, reflections can be reduced using a configuration that does not utilize visual parallax, does not require capturing images of the same scene from multiple viewpoints, and can reduce reflections in scenes other than scenes with a background scene and a foreground scene.

[0101] Other embodiments

[0102] In the above embodiment, the image processing device includes a standard inward-facing camera and a wide-angle inward-facing camera. However, there may be only one inward-facing camera. In this case, the viewing angle can be changed by electronically cropping the captured image.

[0103] Note that for ease of description and understanding, the Figure 6A and 6B The operation described is an example in which a face is not included in the main subject. In the case in which it is unknown whether a face is included in the main subject, for example, the following operation can be performed.

[0104] In step S602, in the case where a face area is included in the video captured by the camera 107 facing outward, the image processing circuit 104 displays an indicator ( ) indicating the face area superimposed on the video for display. Figure 4D ).

[0105] In step S605, the CPU 101 starts capturing video using the wide-angle, inward-facing camera 110. Furthermore, the CPU 101 instructs the image processing circuit 104 to also perform face detection processing on the video captured by the wide-angle, inward-facing camera 110. Furthermore, the CPU 101 instructs the image processing circuit 104 to detect whether there is a reflection in the video captured by the outward-facing camera 107.

[0106] For example, if the video captured by the outward-facing camera 107 includes an area similar to that captured by the wide-angle inward-facing camera 110, the image processing circuit 104 determines that a reflection exists in the video captured by the outward-facing camera 107. Furthermore, if the facial area detected in the video captured by the wide-angle inward-facing camera 110 is included in an area similar to that in the video captured by the wide-angle inward-facing camera 110, the image processing circuit 104 determines that a reflection of the camera person exists. Furthermore, the image processing circuit 104 notifies the CPU 101 of the face detection result, whether a reflection was detected, and whether a reflection of the camera person was detected.

[0107] Then, in step S606, if the image processing circuit 104 has notified the CPU 101 that there is a reflection of the photographer, the CPU 101 executes step S607. If no notification has been received, the CPU 101 executes step S620.

[0108] However, in the operations of step S610, step S611, and step S612, the face detection result and the determination result regarding reflection received from the image processing circuit 104 in step S605 are used.

[0109] Other embodiments

[0110] The embodiments of the present invention can also be implemented by the following method, that is, providing software (program) that performs the functions of the above-mentioned embodiments to a system or device through a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0111] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: one or more processors that execute a program stored in a memory to thereby function as an acquisition unit that acquires first image data obtained by capturing an image of a subject, second image data obtained by capturing an image of a photographer and his or her surroundings, and third image data indicating an appearance of an imaging device that captured the image of the subject; as well as An image processing circuit that uses a learned machine learning model using the first image data, the second image data, and the third image data as input data to obtain image data in which reflection is reduced in the first image data.

2. The image processing apparatus according to claim 1, further comprising: A storage device that stores the learned machine learning model.

3. The image processing apparatus according to claim 1, wherein The image processing circuit uses the learned machine learning model downloaded from an external device.

4. The image processing apparatus according to claim 1, wherein The image processing circuit sends the first image data, the second image data, and the third image data to an external device including the learned machine learning model, and acquires image data with reduced reflection from the external device. The image processing apparatus according to claim 1 , wherein: The learned machine learning model is a machine learning model trained using the first image data, the second image data, and the third image data as input data and using image data without reflection corresponding to the first image data as target data. The image processing apparatus according to claim 1 , wherein: The first image data and the second image data are image data captured by different imaging devices at corresponding timings.

7. The image processing apparatus according to claim 1, further comprising: a first camera device; a second camera device having a different camera direction from that of the first camera device; as well as a storage device storing the third image data, wherein Image data obtained by the first imaging apparatus is used as the first image data, and image data obtained by the second imaging apparatus is used as the second image data.

8. The image processing apparatus according to claim 1, wherein The image processing circuit displays a video based on the first image data and a video based on the second image data on a display device.

9. The image processing apparatus according to claim 1, wherein When the image processing circuit determines that the first image represented by the first image data includes a region similar to the second image represented by the second image data, the image processing circuit determines that reflection exists in the first image data.

10. The image processing apparatus according to claim 7, wherein When the first image represented by the first image data includes an area similar to the second image represented by the second image data, a facial area is detected in the area similar to the second image, and no facial area is detected in the second image, the image processing circuit displays an indicator on the display device, which prompts the user of the image processing device to change the position of the image processing device so that the face is captured by the second camera device. The image processing apparatus according to claim 10 , wherein: The indicator is displayed on the display device superimposed on the video based on the second image data.

12. An image processing method performed by an image processing device, comprising: acquiring first image data obtained by capturing an image of a subject, second image data obtained by capturing an image of a photographer and his / her surroundings, and third image data indicating an appearance of an imaging device that captured the image of the subject; as well as Image data in which reflection in the first image data is reduced is acquired using a learned machine learning model using the first image data, the second image data, and the third image data as input data. 13 . A computer-readable storage medium storing a program for causing a computer to execute the image processing method according to claim 12 .

Images