Image processing device and image processing method
The image processing device uses a smartphone with multiple cameras and a machine learning model to reduce reflections in captured images without requiring multiple viewpoints, improving subject visibility and privacy by minimizing reflections.
Patent Information
- Application Number
- JP2021017739
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-02-05
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2041-02-05
AI Technical Summary
Existing image processing techniques require images to be captured from multiple viewpoints to effectively remove reflections, limiting the flexibility of imaging scenes and methods.
An image processing device and method that utilizes a smartphone with two inward-facing cameras and one outward-facing camera, along with a trained machine learning model, to reduce reflections by capturing images of the subject, photographer, and the device's appearance, allowing for reflections to be minimized without the need for multiple viewpoint capture.
This approach reduces the restrictions on imaging scenes and methods by effectively removing reflections, enhancing visibility of the intended subject and preventing personal information leakage, while eliminating the need for multiple viewpoint imaging.
Smart Images

Figure 0007591937000001 
Figure 0007591937000002 
Figure 0007591937000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing device and an image processing method, and more particularly to a technique for removing reflections in captured images. [Background technology]
[0002] When an image is taken through glass or of a shiny object, the photographer or surrounding objects may be captured in the image. Such capture not only makes it difficult to see the intended subject, but can also lead to the leakage of personal information.
[0003] Non-Patent Document 1 describes a technique for removing reflections in an image captured through glass by utilizing parallax between a distant view and a near view. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] Tianfan Xue, Michael Rubinstein, Ce Liu, William T. Freeman, “A Computational Approach for Obstruction-Free Photography”, ACM Transactions on Graphics, vol. 34, no. 4 (Proc. SIGGRAPH), August 2015 Summary of the Invention [Problem to be solved by the invention]
[0005] However, the technique described in Non-Patent Document 1 has a restriction in that it requires images of a scene containing a distant view and a near view captured from a plurality of viewpoints.
[0006] An object of the present invention is to alleviate restrictions on imaging scenes and imaging methods in an image processing device and image processing method that reduce reflections in captured images. [Means for solving the problem]
[0007] The above-mentioned object can be achieved by an image processing device characterized by having an acquisition means for acquiring first image data obtained by capturing an image of a subject, second image data obtained by capturing an image of the photographer who captured the subject and his / her surroundings, and third image data showing the appearance of the imaging device that captured the subject, and an image processing means for acquiring image data in which reflections in the first image data have been reduced, using a trained machine learning model that uses the first image data, the second image data, and the third image data as input data. Effect of the Invention
[0008] According to the present invention, in an image processing device and an image processing method for reducing reflections in a captured image, restrictions on the captured scene and the capturing method can be reduced. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a smartphone as an example of an image processing device according to an embodiment of the present invention. [Diagram 2] A diagram showing an example of the appearance of the smartphone in Fig. 1. [Diagram 3] A conceptual diagram of a glare reduction process using a trained machine learning model in this embodiment. [Figure 4] 1A and 1C to 1F are diagrams showing a display example of a glare-reducing imaging application according to an embodiment of the present invention, and FIG. 1B is a diagram showing an example of the appearance of a subject; [Diagram 5] FIG. 1 is a diagram showing a display example of a glare-reducing imaging application according to an embodiment of the present invention; [Figure 6] A flowchart showing the operation of a smartphone in a reflection reduction mode according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.
[0011] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in a smartphone equipped with a camera (in-camera) facing inward (toward the photographer) and a camera (out-camera) facing outward (toward the subject). However, the present invention can also be implemented in other electronic devices equipped with an in-camera and an out-camera, such as a tablet terminal or a notebook computer. The in-camera and the out-camera may be detachable external devices.
[0012] Furthermore, the imaging function is not essential to the present invention, and the present invention can be implemented in any electronic device or system that can acquire data on the captured image of the subject and the image of the photographer at the time of capturing the image by some method. For example, the present invention can be implemented in an electronic device that can acquire such image data from an external device that is communicably connected.
[0013] Although it is ideal that the captured image of the subject and the image of the photographer at the time of capturing images are captured at the same timing, the capture timing may be different as long as it is a short time (for example, within one second, within a few seconds, etc.) Here, the same timing may mean that the capture instruction is output to the imaging device capturing each image substantially at the same time, or the exposure periods for capturing images overlap.
[0014] FIG. 1 is a diagram illustrating an example of the configuration of a smartphone 100 as an example of an image processing device according to the present invention, using functional blocks. The internal bus 130 has data signal lines, control signal lines, etc., and realizes communication of data and control signals between the connected functional blocks.
[0015] The CPU 101 is a main control unit, and controls the operation of each functional block by, for example, loading programs (such as an OS and applications) stored in the non-volatile memory 103 into the memory 102 and executing them, thereby implementing various functions of the smartphone 100.
[0016] The memory 102 is, for example, a RAM, and is used as a main storage device, a work memory, a buffer memory, a video memory, and the like.
[0017] The non-volatile memory 103 is, for example, an electrically rewritable ROM. The non-volatile memory 103 stores system data such as programs executed by the CPU 101, various setting values, and GUI data, as well as user data such as still and video data captured by the user, and applications.
[0018] Under the control of the CPU 101, the image processing unit 104 applies various image processing to image data captured by the outer camera 107, the standard inner camera 109, and the wide-angle inner camera 110, or to image data stored in the non-volatile memory 103. The image processing unit 104 may be a dedicated hardware circuit such as an ASIC designed to realize a specific function. Department The image processing unit 104 may be configured to realize a specific function by a programmable processor such as a DSP executing a program stored in the non-volatile memory 103. The image processing unit 104 may also include a dedicated circuit for quickly executing calculations required for machine learning such as deep learning. Depending on the content of the processing, the CPU 101 may execute a program to apply image processing instead of the image processing unit 104.
[0019] Here, the image processing applied by the image processing unit 104 includes pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing includes noise reduction, defective pixel correction, A / D conversion, and D / A conversion. Color interpolation processing is processing for interpolating values of color components that cannot be obtained during imaging, and is also called demosaic processing or synchronization processing. The correction processing includes white balance adjustment, gradation correction (gamma processing), processing to correct the effects of optical aberration and peripheral light falloff in the optical system, and color correction processing. The detection process includes detection of characteristic regions (for example, face regions and human body regions) and their movements, and person recognition processing. The data processing includes synthesis, scaling, encoding and decoding, header information generation, and the like. The evaluation value calculation process includes the generation of signals and evaluation values used in automatic focus detection (AF), and the calculation of evaluation values used in automatic exposure control (AE). The special effect processing includes adding blurring, changing color tones, relighting processing, etc. Note that these are examples of image processing that can be applied by the image processing unit 104, and do not limit the image processing that the image processing unit 104 applies.
[0020] The display 105 is, for example, a touch-sensitive liquid crystal display. Under the control of the CPU 101, the display 105 displays GUI (Graphical User Interface) images (icons, windows, software keys, etc.) of the OS and applications, images captured by a camera, and the like.
[0021] The CPU 101 generates display data and stores it in a video memory area of the memory 102. The display 105 generates a video signal based on the display data and displays it on a display panel. The video signal may also be supplied to an external display.
[0022] Operation unit 106 is a general term for components that can be operated by the user, such as buttons, switches (hardware or software), and a touch panel. CPU 101 monitors operation unit 106 and executes an operation according to the operation target and operation content. For touch operations on display 105, CPU 101 detects the touch position and recognizes predetermined touch operations (tap, drag, pinch, swipe, etc.). CPU 101 also detects operations on input devices such as an externally connected keyboard and mouse.
[0023] Smartphone 100 has one outer camera 107 and two inner cameras 109 and 110. The two inner cameras 109 and 110 have fixed-focus lenses with different angles of view, and the lens of inner camera 110 has a wider angle of view than the lens of inner camera 109. For this reason, in this specification, they are referred to as standard inner camera 109 and wide-angle inner camera 110. However, standard and wide angle refer to the relationship between the angles of view of the two inner cameras, and do not refer to any specific angle of view.
[0024] The outer camera 107 is a camera unit having an imaging element such as a CCD / CMOS image sensor, an imaging optical system (lens), a shutter, an aperture, a focus detection circuit, an A / D converter, and the like. The lens of the outer camera 107 may be a zoom lens or a single focus lens. The lens also includes a focus lens. The outer camera 107 is capable of capturing still images and moving images. Image data obtained by capturing an image is transmitted to the image processing unit 104, stored in a data file of a predetermined format, and recorded in the non-volatile memory 103. If the display surface of the display 105 is considered to be the front surface of the smartphone 100, the outer camera 107 is provided on the back surface of the smartphone 100.
[0025] The light emitting unit 108 is, for example, an auxiliary light source having one or more light emitting elements (for example, LEDs). It is used as a flash or video light when capturing an image. The light emitting unit 108 may be capable of adjusting the color of the light.
[0026] Standard in-camera 109 and wide-angle in-camera 110 are camera units having a similar configuration to out-camera 107 having a single focus lens. Standard in-camera 109 and wide-angle in-camera 110 are provided on the surface of smartphone 100. Therefore, the imaging directions of out-camera 107 and the two in-cameras 109 and 110 differ by 180°.
[0027] The communication I / F 111 is an interface for communicating with an external device connected to the communication I / F 111 so as to be communicable with the communication I / F 111 via a wired or wireless connection. The communication I / F 111 can support a plurality of communication standards. The external device may be connected to the communication I / F 111 directly or via a network 112.
[0028] Fig. 2 shows an example of the appearance of the smartphone 100, and Fig. 2(a) is a plan view of the surface on which the display screen of the display 105 is present (hereinafter, referred to as the front surface or display surface). Fig. 2(b) is a plan view of the surface (back surface) parallel to the front surface. In Fig. 2, the components shown in Fig. 1 are given the same reference numerals as in Fig. 1.
[0029] The smartphone 100 of this embodiment can set a reflection reduction mode when capturing an image using the outer camera 107. When the reflection reduction mode is set, one of the inner cameras (for example, the wide-angle inner camera 110) also captures an image in parallel with capturing an image using the outer camera 107. Then, the image of the photographer captured by the wide-angle inner camera 110 is used to reduce reflection in the image of the subject captured by the outer camera 107.
[0030] The reflection reduction mode can be set by the user through the operation of a GUI included in the screen of an application capable of capturing images using the outer camera 107 and the inner camera 110. Alternatively, the reflection reduction mode may be set as a default for a specific application when the application is started. Also, the image processing unit 104 may analyze a moving image captured for EVF display in an imaging standby state, and the reflection reduction mode may be automatically set when reflection is detected.
[0031] In the following, as an example, the reflection reduction process is described as being executed by an application for capturing images in the reflection reduction mode. However, the reflection reduction process may be implemented as one function of a general-purpose camera application, or may be a function provided by the OS.
[0032] As an example, a trained machine learning model implemented by a neural network is used for the reflection reduction process for the image captured by the outer camera 107. FIG. 3 is a conceptual diagram of the reflection reduction process using a trained machine learning model 302.
[0033] Machine learning model 302 is First image data (subject image) obtained by capturing an image of a subject with the outer camera 107; Second image data (image of the photographer) obtained by capturing an image of the photographer who captured the subject and his / her surroundings using the wide-angle inner camera 110; Third image data (device image) showing the appearance of an imaging device (here, the smartphone 100) that captured the subject; The external image of the smartphone 100 is an image captured from the rear of the smartphone 100, and can be stored in the non-volatile memory 103 in advance, for example.
[0034] The machine learning model 302 may be stored in the non-volatile memory 103, or may be included in an external device with which the smartphone 100 can communicate. When the machine learning model 302 exists in an external device, it may be downloaded to the smartphone 100 via the communication I / F 111 for use. Instead of downloading, the input data 301 may be transmitted to the external device, and the calculation of the machine learning model 302 may be performed by the external device. When the calculation of the machine learning model 302 is performed by an external device, the smartphone 100 receives the calculation result (a subject image with reduced reflection) from the external device.
[0035] Here, it is assumed that the learning model 302 exists in the smartphone 100, either by being stored in the non-volatile memory 103 or by being downloaded from an external device.
[0036] The implementation method (neural network configuration, number of layers, activation function, etc.) and learning method (loss function, optimization algorithm, etc.) of the machine learning model 302 can use known techniques. Furthermore, the present invention does not depend on the implementation method or learning method of the machine learning model 302. Therefore, here, only an example of the learning method of the machine learning model 302 will be briefly described, and other details will be omitted.
[0037] The machine learning model 302 can learn using supervised learning. Specifically, a subject image with reflection captured by the outer camera 107, a photographer image captured by the wide-angle inner camera 110, and a device image of the smartphone 100 are used as input data. In addition, an image without reflection or with reduced reflection that corresponds to the subject image of the input data is used as the teacher data. The teacher data may be generated by applying image processing to the subject image with reflection, or a subject image captured from the same viewpoint as the subject image with reflection so that the reflection is not present may be used. For example, in the case of imaging through glass, an image with reflection can be obtained by capturing the image normally, and an image without reflection can be obtained by capturing the image with light shielded around the smartphone 100.
[0038] By preparing input data and learning data for various subjects and imaging environments and learning the machine learning model, a trained machine learning model 302 can be obtained.
[0039] Here, we will explain why wide-angle in-camera images are provided to the machine learning model. imageWhat is reflected in the image corresponds to a mirror image of the image captured in the direction of the outer camera 107 from the subject. The image captured in the direction of the outer camera 107 from the subject has significant similarity to the image captured by the inner camera. Based on the similarity between the subject image captured by the outer camera 107 and the image of the photographer and his / her surroundings captured by the inner camera, it is possible to converge the learning of the machine learning model so that the output result approaches an image from which reflections have been removed or reduced, which is the training data.
[0040] Furthermore, an image obtained by the wide-angle inner camera 110, which can capture a wider range of the surroundings of the photographer than the standard inner camera 109, is more advantageous in detecting similarity with the captured image than an image obtained by the standard inner camera 109.
[0041] The reason why the exterior image of the smartphone 100 is provided to the machine learning model is that the smartphone 100 that captured the subject is likely to appear in the subject image. By using the exterior image of the smartphone 100 as input data, the machine learning model can easily detect the smartphone 100 that appears in the subject image, which is advantageous in converging learning so that the output result approaches the teacher data.
[0042] 4 and 5 are diagrams showing display examples of the glare reducing application. The application is executed by the CPU 101, and the display is implemented by controlling the image processing unit 104 and the like.
[0043] 4(a) shows the display state when the application is started. A GUI of an image capture button 401 is displayed at the bottom of the application screen. When the application is started, the application enters an image capture standby state, and a video 402 captured by the outer camera 107 is displayed in real time on the application screen. The CPU 101 continuously executes the video capture process by the outer camera 107, the process of generating video data for display by the image processing unit 104, and the process of displaying the video data by the display 105.
[0044] The user can check the imaging range, exposure state, and the like from the moving image 402. Then, when capturing an image, the user touches the imaging button 401. When the touch operation of the imaging button 401 is detected, the CPU 101 executes imaging processing in the reflection reduction mode.
[0045] FIG. 4(a) shows a scene in which the TV is the main subject, with the user and his / her surroundings reflected on the TV screen. FIG. 4(b) shows an image in which the reflection has been removed by reflection reduction processing. By reducing the reflection in this way, an image in which the main subject is easy to see can be obtained. In addition, even if the image is uploaded to the Internet, it is possible to prevent the user's personal information from being leaked from the image.
[0046] Furthermore, in the reflection reduction mode, CPU 101 also captures moving images with wide-angle in-camera 110 in standby state. Then, image processing unit 104 detects whether or not a user is captured from the image captured by in-camera 110 and the image captured by out-camera 107. Image processing unit 104 uses a known technique such as pattern matching to detect whether or not a user is captured, and notifies CPU 101 of the result.
[0047] When CPU 101 is notified by image processing unit 104 that the user is captured in the image, it instructs image processing unit 104 to perform, for example, picture-in-picture display of the moving image captured by wide-angle inner camera 110. In response to the instruction, image processing unit 104 starts picture-in-picture display of the moving image captured by wide-angle inner camera 110 (FIG. 4(c)).
[0048] Furthermore, image processing unit 104 superimposes an index on the moving image captured by outer camera 107 to inform the user of the face being captured. Fig. 4(d) shows an example in which a frame-shaped index is superimposed on an area in which the user's face is captured. Image processing unit 104 detects an area in the moving image captured by outer camera 107 that is similar to the face area detected in the moving image captured by wide-angle inner camera 110, and superimposes a frame image around it.
[0049] In addition, the image processing unit 104 also displays an indicator in the video captured by the wide-angle in-camera 110, indicating an area similar to a face captured in the video captured by the out-camera 107 (Figure 4(e)).
[0050] This makes it possible to inform the user that by using the face shown in the image captured by the wide-angle inner camera 110, it is highly possible to reduce the appearance of the face in the image captured by the outer camera 107.
[0051] On the other hand, there are cases where image processing unit 104 determines that the image captured by outer camera 107 contains an object including a face, but the image captured by wide-angle inner camera 110 does not contain the face of the person. In such a case, image processing unit 104 displays an indicator on display 105 to prompt the user to have the user's face captured by wide-angle inner camera 110.
[0052] As an indicator, image processing unit 104 superimposes a frame and a figure resembling a face on the image captured by wide-angle inner camera 110, as shown in Fig. 10, for example. Note that it is also possible to determine where and at what size the user's face should be captured in the shooting range of wide-angle inner camera 110 based on the captured image, and to display the indicator at a position and size according to the determination result. Note that the indicator may be a message showing specific instructions in writing. By displaying such an indicator, it is possible to increase the possibility that the capture reduction can be more appropriately implemented when a face is captured in the image captured by outer camera 107.
[0053] Even if such an indicator is displayed, the image capture button 401 may be operated when the face is not captured by the wide-angle inner camera 110. In such a case, the CPU 101 instructs the image processing unit 104 to display a message on the display 105 to inform the user that the capture of the face cannot be reduced. Figure 5(a) is an example of the message.
[0054] When a user operation, for example, a tap operation on the display 105, is detected in the state of Fig. 5(a), the CPU 101 instructs the image processing unit 104 to display a dialog box on the display 105 to confirm with the user whether or not to continue capturing images. Fig. 5(b) is an example of the confirmation dialog box.
[0055] The confirmation dialog includes a "Yes" button and a "No" button for instructing whether or not to continue capturing an image. When CPU 101 detects a touch operation of the "Yes" button, it starts capturing an image of a still image. When CPU 101 detects a touch operation of the "No" button, it notifies image processing unit 104 to erase the confirmation dialog. In this case, the display of the reflection reduction application returns to the state shown in FIG. 4(f).
[0056] Next, the operation of the CPU 101 regarding image capture in the reflection reduction mode will be described with reference to the flowchart shown in Fig. 6. Here, the image capture environment shown in Fig. 4 is assumed. In S601, the CPU 101 determines whether or not an operation to turn on the reflection-reduced imaging mode by the user has been detected. Here, the operation to start the reflection-reduced imaging application installed in the smartphone 100 is the operation to turn on the reflection-reduced imaging mode. Note that instead of detecting the operation, it may be determined whether or not the current shooting mode is set to the reflection-reduced mode. If it is determined that an operation to turn on the reflection-reduced imaging mode by the user has been detected, the CPU 101 executes S602, and if it is not determined that an operation to turn on the reflection-reduced imaging mode by the user has been detected, the CPU 101 repeatedly executes S601.
[0057] In S602, the CPU 101 starts operation in a shooting standby state. The CPU 101 starts capturing moving images with the outer camera 107. The CPU 101 also instructs the image processing unit 104 to generate a moving image for display and to display the generated moving image. As a result, a moving image showing the imaging range of the outer camera 107 is continuously displayed on the display 105 (FIG. 4(a)).
[0058] Furthermore, CPU 101 instructs image processing unit 104 to execute face detection processing on the moving image captured by rear camera 107. Image processing unit 104 starts face detection processing on frame images of the moving image, and notifies CPU 101 of the detection results (number, size, position, etc. of detected face areas). The frequency of executing face detection processing can be determined according to the frame rate and performance of image processing unit 104. Image processing unit 104 may execute face detection processing continuously, or may execute it on one frame each time it is instructed by CPU 101. Then, CPU 101 executes S603.
[0059] In S603, CPU 101 determines whether the angle of view of the in-camera is set to wide-angle. CPU 101 reads the current angle of view setting of the in-camera from memory 102, for example, and determines whether the setting is set to use wide-angle in-camera 110. Alternatively, CPU 101 may determine whether wide-angle in-camera 110 is valid out of standard in-camera 109 and wide-angle in-camera 110. If CPU 101 determines that the angle of view setting of the in-camera is set to wide-angle, it executes S605, and if not, it executes S604.
[0060] In S604, the CPU 101 changes the angle of view setting of the in-camera stored in the memory 102 to a wide angle. After that, the CPU 101 executes S605.
[0061] In S605, the CPU 101 analyzes the moving image captured by the out-camera 107 in S602. For example, the CPU 101 instructs the image processing unit 104 to execute face detection processing on the moving image captured by the out-camera 107. Note that if the image processing unit 104 has been instructed to continue executing face detection processing in S602, S605 may be omitted. Thereafter, the CPU 101 executes S606.
[0062] In S606, the CPU 101 determines whether or not a face is captured in the moving image captured by the outer camera 107. For example, the CPU 101 acquires the result of face detection processing by the image processing unit 104, and if a face area is detected, it determines that a face is captured and executes S607, whereas if a face area is not detected, it determines that a face is not captured and executes S620.
[0063] In S607, CPU 101 starts capturing moving images with wide-angle inner camera 110 (it may start in S605). CPU 101 also instructs image processing unit 104 to generate a moving image for display from the moving image captured by wide-angle inner camera 110, and to perform picture-in-picture display of the generated moving image. As a result, a moving image showing the imaging range of outer camera 107 and a moving image captured by wide-angle inner camera 110 are continuously displayed on display 105 (FIG. 4(c)). Thereafter, CPU 101 executes S608.
[0064] In S608, CPU 101 instructs image processing unit 104 to superimpose an index indicating the detected face area on the moving image captured by out-camera 107. As a result, as shown in Fig. 4(d), an index indicating the detected face area is superimposed on the moving image captured by out-camera 107. Note that the superimposed display of the index indicating the face area may be instructed at the same time as instructing detection of the face area. Thereafter, CPU 101 executes S609.
[0065] In S609, the CPU 101 instructs the image processing unit 104 to apply face detection processing to the moving image captured by the wide-angle inner camera 110. Then, the CPU 101 executes S610. Note that the processing of S609 may be included in the processing of S607.
[0066] In S610, CPU 101 determines whether or not a face is included in the moving image captured by wide-angle inner camera 110 based on the result of face detection processing notified by image processing unit 104. CPU 101 executes S611 if a face is detected in the moving image captured by wide-angle inner camera 110, and executes S614 if a face is not detected.
[0067] In S611, CPU 101 instructs image processor 104 to calculate a degree of matching indicating the degree to which a face detected in a moving image captured by out-camera 107 matches a face detected in a moving image captured by wide-angle in-camera 110. Image processor 104 quantifies facial features such as the positions, shapes, and sizes of eyes, nose, mouth, etc., the overall shape of the face, and unevenness from the face areas detected in each moving image. Image processor 104 then compares the numerical values related to the facial features to calculate the degree of matching between the faces. The degree of matching between the faces may be, for example, the sum of the differences in the numerical values, and in this case, the smaller the value, the higher the degree of matching between the faces. Image processor 104 notifies CPU 101 of the calculated degree of matching between the faces. When CPU 101 is notified of the degree of matching between the faces, it executes S612.
[0068] In S612, CPU 101 determines whether the face detected in the video captured by out-camera 107 and the face detected in the video captured by wide-angle in-camera 110 are the face of the same person based on the degree of face matching. For example, if the degree of face matching is less than a threshold, CPU 101 determines that they are the faces of the same person and executes S613, and if the degree of face matching is equal to or greater than the threshold, CPU 101 determines that they are the faces of different people and executes S615.
[0069] The processes of S611 and S612 may be realized by inference processing using a trained model. In this case, image data of the face part detected in the video captured by the outer camera 107 and image data of the face part detected in the video captured by the wide-angle inner camera 110 are used as input data of the trained model. Then, the trained model outputs the degree of match as output data. In addition, such a trained model can be generated by machine learning using, for example, a neural network. In this case, the machine learning may be supervised learning in which image data of the face part captured in advance by the outer camera 107 and image data of the face part captured by the wide-angle inner camera 110 are used as input data, and the degree of match between the two obtained by a predetermined method is used as supervised data.
[0070] In S613, CPU 101 instructs image processing unit 104 to superimpose an index indicating the detected face detection on the moving image captured by wide-angle inner camera 110. As a result, as shown in Fig. 4(e), an index indicating the detected face area is superimposed on the moving image captured by wide-angle inner camera 110. Note that the superimposed display of the index indicating the face area may be instructed at the same time as instructing detection of the face area. Thereafter, CPU 101 executes S620.
[0071] In S614, CPU 101 instructs image processing unit 104 to display an indicator (face position guide) that prompts the user to change his / her position so that the user's face is captured by wide-angle inner camera 110. In response to the instruction, image processing unit 104 superimposes a frame and a figure resembling a face in the center of the moving image captured by wide-angle inner camera 110 as shown in Fig. 4(f), or superimposes a message in the form shown in Fig. 5(a), for example. CPU 101 then executes S615.
[0072] In S615, the CPU 101 determines whether or not a user has performed an image capturing operation. Specifically, the CPU 101 determines whether or not a touch operation has been performed on the image capturing button 401. If it is determined that an image capturing operation has been performed, the CPU 101 executes S616, and if not, the CPU 101 executes S605.
[0073] In S616, CPU 101 instructs image processing unit 104 to superimpose a message indicating that the reflection of the face cannot be reduced. Here, image processing unit 104 displays a message on display 105 saying "The photographer may be reflected in the subject. The photographer is not recognized by the in-camera, so the reflection cannot be reduced" (FIG. 5(a)). Then, CPU 101 executes S617.
[0074] In S617, the CPU 101 determines whether or not a confirmation operation for the message, for example, a tap operation on the display 105, has been detected. If the CPU 101 determines that a confirmation operation has been performed, it instructs the image processing unit 104 to display on the display 105 a dialog for confirming with the user whether or not to continue capturing images. Here, the image processing unit 104 displays on the display 105 a message saying "Do you want to continue capturing images?" and a dialog including "Yes" and "No" buttons (FIG. 5(b)). Note that instead of detecting a confirmation operation, it may be determined whether or not a predetermined time has elapsed. Thereafter, the CPU 101 executes S618.
[0075] In S618, the CPU 101 determines whether or not an image capture cancel operation has been detected. Specifically, the CPU 101 determines whether or not an operation on the "No" button in the confirmation dialog displayed in S617 has been detected. If it is determined that an image capture cancel operation has been detected, the CPU 101 executes S605, and if not, the CPU 101 executes S619.
[0076] In S619, the CPU 101 determines whether or not an image capture continuation operation has been detected. Specifically, the CPU 101 determines whether or not an operation on the “Yes” button of the confirmation message displayed in S617 has been detected. If it is determined that an image capture continuation operation has been detected, the CPU 101 executes S621, and if not, the CPU 101 executes S618.
[0077] In S620, the CPU 101 determines whether or not the user has performed an image capturing operation, similarly to S615. If it is determined that an image capturing operation has been performed, the CPU 101 executes S621, and if not, the CPU 101 executes S605.
[0078] In S621, CPU 101 captures still images using each of outer camera 107 and wide-angle inner camera 110. Parameters such as aperture and shutter speed used for capturing still images are determined by CPU 101 based on an evaluation value generated by image processing unit 104 based on a moving image captured by outer camera 107. CPU 101 also performs autofocus detection of the optical system based on an evaluation value generated by image processing unit 104 based on a moving image captured by outer camera 107.
[0079] Furthermore, CPU 101 instructs image processing unit 104 to generate still image data for recording based on still image data captured by outer camera 107, and still image data for use in a reflection reduction mode based on still image data acquired by wide-angle inner camera 110. Image processing unit 104 applies image processing to the still image data, prioritizing image quality over image processing for generating moving image data for display. Then, CPU 101 executes S622.
[0080] In S622, CPU 101 instructs image processing unit 104 to apply reflection reduction processing to the still image data captured by outer camera 107 in S621. Image processing unit 104 reads out trained machine learning model 302 and an exterior image of smartphone 100 from non-volatile memory 103, and reads them into internal memory or memory 102. After that, image processing unit 104 provides still image data for recording, still image data generated from still image data captured by wide-angle inner camera 110, and the exterior image of smartphone 100 as input data to trained machine learning model 302. Image processing unit 104 performs calculations of trained machine learning model 302, and stores the image with reduced reflection obtained as an output in non-volatile memory 103.
[0081] In this way, the image processing device according to the present embodiment can reduce reflections in the subject image by providing the image of the subject, the image of the photographer and its surroundings, and the image of the exterior of the imaging device as input data to the trained machine learning model. Unlike a configuration using parallax, by using the image of the photographer and its surroundings, and the image of the exterior of the imaging device, it is not necessary to capture the same scene from multiple viewpoints, and reflections can be reduced even in scenes other than those in which a distant view and a close-up view exist.
[0082] (Other embodiments) In the above embodiment, the image processing device has a standard in-camera and a wide-angle in-camera. However, the image processing device may have only one in-camera. In this case, the image processing device may be configured to change the angle of view by electronically cropping the captured image.
[0083] For ease of explanation and understanding, the operation when it is known that the main subject does not include a face has been described in Fig. 6. When it is not known whether the main subject includes a face, the operation will be as follows, for example.
[0084] In S602, if a face region is included in the moving image captured by the outer camera 107, the image processing unit 104 superimposes an indicator indicating the face region on the moving image for display (FIG. 4(d)).
[0085] In S605, CPU 101 starts capturing moving images with wide-angle inner camera 110. CPU 101 also instructs image processing unit 104 to execute face detection processing on the moving images captured by wide-angle inner camera 110. CPU 101 further instructs image processing unit 104 to detect whether or not any object is captured in the moving images captured by outer camera 107.
[0086] For example, if the moving image captured by outer camera 107 contains an area similar to the moving image captured by wide-angle inner camera 110, image processing unit 104 determines that the moving image captured by outer camera 107 contains an object. Also, image processing unit 104 determines that the photographer is captured if a face area detected in the moving image captured by wide-angle inner camera 110 is included in an area in the moving image captured by outer camera 107 similar to the moving image captured by wide-angle inner camera 110. Then, image processing unit 104 notifies CPU 101 of the face detection result, whether or not an object is detected, and whether or not the photographer is detected.
[0087] Then, in S606, if the CPU 101 has received a notification from the image processing unit 104 that the photographer is included in the image, the CPU 101 executes S607, and if not, the CPU 101 executes S620.
[0088] The operations from S607 onward may be performed as described above, except that the operations of S610, S611, and S612 use the face detection result and the determination result regarding reflection notified by the image processing unit 104 in S605.
[0089] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
[0090] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]
[0091] 101: CPU, 102: memory, 103: non-volatile memory, 104: image processing unit, 105: display, 106: operation unit, 107: outer camera, 109: standard inner camera, 110: wide-angle inner camera
Claims
1. an acquisition means for acquiring first image data obtained by capturing an image of a subject, second image data obtained by capturing an image of a person who captured the image of the subject and his / her surroundings, and third image data showing an external appearance of an image capturing device that captured the image of the subject; An image processing means for acquiring image data in which the reflection in the first image data is reduced by using a trained machine learning model having the first image data, the second image data, and the third image data as input data; 13. An image processing device comprising:
2. The image processing device according to claim 1 , further comprising a storage means for storing the trained machine learning model.
3. The image processing device according to claim 1 , wherein the image processing means downloads the trained machine learning model from an external device and uses the trained machine learning model.
4. The image processing device according to claim 1, characterized in that the image processing means transmits the first image data, the second image data, and the third image data to an external device having the trained machine learning model, and acquires the image data with the reflection reduced from the external device.
5. 5. The image processing device according to claim 1, wherein the trained machine learning model is a machine learning model trained using the first image data, the second image data, and the third image data as input data and image data without reflections corresponding to the first image data as training data.
6. 6. The image processing apparatus according to claim 1, wherein the first image data and the second image data are image data captured at the same time by different image capturing devices.
7. A first imaging means; a second imaging means having an imaging direction different from that of the first imaging means; and a storage means for storing the third image data. Image data obtained by the first imaging means is used as the first image data, and image data obtained by the second imaging means is used as the second image data.
7. The image processing device according to claim 1, wherein the image processing device further comprises: a first input unit;
8. 8. The image processing apparatus according to claim 1, wherein the image processing means causes a display device to display a moving image based on the first image data and a moving image based on the second image data.
9. The image processing device according to any one of claims 1 to 8, characterized in that the image processing means determines that the first image data has a reflection when it is determined that the first image represented by the first image data contains an area similar to the second image represented by the second image data.
10. The image processing device according to claim 7, characterized in that, when a first image represented by the first image data includes an area similar to a second image represented by the second image data, a facial area is detected in the similar area, and a facial area is not detected in the second image, the image processing means displays an indicator on a display device prompting a user of the image processing device to change the position so that a face is captured by the second imaging means.
11. 11. The image processing apparatus according to claim 10, wherein the indicator is displayed on a display device so as to be superimposed on a moving image based on the second image data.
12. An image processing method executed by an image processing device, comprising: Obtaining first image data obtained by imaging a subject, second image data obtained by imaging a person who images the subject and his / her surroundings, and third image data showing an external appearance of an imaging device that images the subject; Obtaining image data in which the reflection in the first image data is reduced using a trained machine learning model that uses the first image data, the second image data, and the third image data as input data; and 13. An image processing method comprising:
13. A program for causing a computer to function as the acquisition unit and the image processing unit of the image processing device according to any one of claims 1 to 11.
Citation Information
Patent Citations
Face image processing device
JP2009169740A
Digital camera, corrected image generating method and storage medium
JP2010074693A
Image processing apparatus, method therefor, program, and image capturing apparatus
JP2012043151A
Image pickup device
WO2018234925A1
