IMAGE PROCESSING DEVICE, CAMERA DEVICE AND IMAGE PROCESSING METHOD

Description

BACKGROUND OF THE INVENTION1. Field of the Invention

[0001] A preferred embodiment of the present invention relates to an image processing apparatus, a camera apparatus, and an image processing method that process an image captured by a camera.2. Description of the Related Art

[0002] In a case in which a teleconference is performed by connecting remote places through a network, a plurality of conference participants may be captured by one camera. However, when the plurality of conference participants are captured by one camera, a face image of each conference participant is small. Therefore, the conference participants can hardly read facial expression of other conference participants.

[0003] Thus, for example, International Publication No. 2018 / 061173 discloses that an image of each conference participant is extracted from an image obtained by capturing a plurality of conference participants at the same time.

[0004] In addition, Japanese Unexamined Patent Application Publication No. 2011-61314 discloses a configuration in which a plurality of places, a plurality of conference participants, and materials are displayed on one screen.

[0005] In addition, Japanese Unexamined Patent Application Publication No. 2015-19162 discloses a configuration in which an image captured by a fisheye lens is converted to a planar image.

[0006] US 2018 / 0262716 A1 discloses a method of providing a video conference service and apparatuses performing the same, the method including determining contributions of a plurality of participants to a video conference based on first video signals and first audio signals of devices of the plurality of participants participating in the video conference, and generating a second video signal and a second audio signal to be transmitted to the devices of the plurality of participants based on the contributions.

[0007] US 6, 297,846 B1 discloses a display control system for a videoconference terminal capable of displaying proper images of participants. A video camera captures wide-angle source pictures including the images of participants. The source pictures are sent to a participant position detector and image clipping unit. The participant position detector detects a participant position by examining the source pictures. Depending on the on-screen movement observed in the participant position detected by the participant position detector, the image clipping unit determines an appropriate clipping window. It clips the desired image out of the source pictures as defined by the clipping window. A screen adjustment unit adjusts the image size so that it will fit to the actual screen size.

[0008] US 2014 / 0098174 A1 discloses methods and systems of receiving one or more images by one or more cameras. Each of the one or more images is acquired by one or more cameras in which the one or more images comprise facial images corresponding to persons. In one embodiment, aspects of the disclosure describe a method for extracting each of the facial images from each of the images in which each of the facial images corresponds to each of one or more sets of extracted facial images. The method further includes sorting each of the extracted facial images per each set into separate groups of one or more groups wherein each group corresponds to facial images of each person. The method further includes selecting a preferred facial image from each group of the one or more groups to generate preferred facial images for transmission to a client using a display server.

[0009] US 2014 / 0376785 A1 discloses a system for enhancing a facial expression which includes a processing circuit configured to receive video of a user, generate facial data corresponding to a face of the user, analyze the facial data to identify a facial expression, enhance the facial data based on the facial expression, and output modified video including the enhanced facial data.SUMMARY OF THE INVENTION

[0010] At a conference, a conference participant can perform smooth communication by reading facial expression and emotion of another conference participant. However, it is difficult to read a feeling of each conference participant from an image that displays a plurality of conference participants.

[0011] In view of the foregoing, a preferred embodiment of the present invention is to provide an image processing apparatus, a camera apparatus, and an image processing method that are configured to enable a conference participant to intuitively read a feeling of each conference participant from an image that displays a plurality of conference participants.

[0012] In particular, the present invention provides an image processing apparatus, in accordance with claim 1.

[0013] The present invention also provides a camera apparatus in accordance with claim 4.

[0014] The present invention also provides an image processing method, in accordance with claim 5.

[0015] According to a preferred embodiment of the present invention, a conference participant is able to intuitively read a feeling of each conference participant from an image that displays a plurality of conference participants.

[0016] The above and other elements, features, steps, characteristics and advantages of the present invention will become more apparent from the following detailed description of the preferred embodiments with reference to the attached drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG. 1 is a block diagram showing a configuration of a camera apparatus including an image processing apparatus. FIG. 2 is a flow chart showing an operation of the image processing apparatus. FIG. 3 is a block diagram showing an image processing function (the image processing apparatus). FIG. 4 is a diagram showing an example of an image that a camera 11 has captured. FIG. 5 is a diagram showing an example of a reorganized image. FIG. 6 is a diagram showing a flow of processing of each of a plurality of image processing apparatuses that configure a teleconference. FIG. 7 is a flow chart showing an operation of the image processing apparatus. FIG. 8 is a diagram showing an example of displaying content related to a conference, together with a face image, in an image. FIG. 9 is a block diagram showing the image processing function further including a facial expression estimating portion, a feeling estimating portion, and a display mode changing portion. FIG. 10 is a flow chart showing an operation of the image processing apparatus. FIG. 11 is a diagram showing an example of changing a display mode. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] A preferred embodiment of the present invention is to provide an image processing apparatus that achieves a smooth conference.

[0019] The present invention provides an image processing apparatus, in accordance with claim 1.

[0020] The image processing apparatus according to the present preferred embodiment of the present invention enables a conference participant to easily read facial expression of each conference participant by extracting a face image of each conference participant from an image obtained by capturing a plurality of conference participants at the same time. Furthermore, since the image processing apparatus changes the face image of each conference participant, depending on the feeling of each conference participant, a conference participant can easily read the feeling of each conference participant more intuitively. For example, the image processing apparatus displays a face image of a conference participant who has a pleasant feeling, in a shaking manner. Another participant who sees the face image changing to shake can intuitively recognize that the conference participant corresponding to the face image has a pleasant feeling. It is to be noted that the display mode changing portion may change a display position of the face image in the one image, based on the estimated feeling. Alternatively,

[0021] the display mode changing portion may add animation to the face image, based on the estimated feeling. In addition, the display mode changing portion may change a color of the face image in the one image, based on the estimated feeling. It is to be noted that a change in color includes a change in hue, a change in brightness, a change in saturation, and a change in tone.

[0022] In addition, the image processing apparatus includes a facial expression estimating portion that estimates facial expression of each conference participant, based on the detected face image. The feeling estimating portion estimates the feeling based on the facial expression estimated by the facial expression estimating portion. Facial expression is an index for expressing a feeling. Facial expressions and feelings are closely related to each other. Therefore, the image processing apparatus, by first estimating facial expression from a face image and then estimating a feeling based on the estimated facial expression, is able to estimate a feeling more accurately.

[0023] In addition, the one image may display content (a shared document, for example) related to a conference, together with the face image.

[0024] The conference participants include a speaker and a listener. The speaker converses using content such as presentation material. Such content is displayed, for example, on a screen for a projector, or a monitor. The speaker and the listener have a conference while looking at the screen or the monitor. The listener, since looking at the screen, has difficulty simultaneously looking at the facial expression of the speaker who is present in the same room, and the content. The speaker also has difficulty simultaneously looking at the facial expression of the listener who is present in the same room, and the content.

[0025] However, as described above, the image processing apparatus reorganizes the face image of each conference participant and the content, into one image. Therefore, the conference participants, since being able to converse while looking at mutual facial expression, can have a smooth conference.

[0026] In addition, the one image may be shared by all of a plurality of image processing apparatuses that configure a teleconference. In other words, the same image may be displayed at all locations. Typically, in a teleconference, an image on a far end side is displayed large and an image on a near end side is displayed small. In such a case, the conference participants are strongly conscious of conversing between one location and another location. However, the image processing apparatus according to the present preferred embodiment of the present invention shares in all locations one image reorganized by extracting a face image of each conference participant, and displays the same image in all the locations. Therefore, the conference participants can easily recognize that all the conference participants are present in the same place and having a conference, without being conscious of a far end side or a near end side.

[0027] Hereinafter, a specific configuration according to a preferred embodiment of the present invention will be described in relation to Figures 9 to 11. Background examples useful for understanding the invention are described in relation to Figures 1 to 8.

[0028] FIG. 1 is a block diagram showing a configuration of a camera apparatus 1 including an image processing apparatus 10. FIG. 2 is a flow chart showing an operation of the image processing apparatus 10.

[0029] The camera apparatus 1 includes a camera 11, a CPU 12, a DSP 13, a memory 14, a user interface (UI) 15, a speaker (SP) 16, a microphone (MIC) 17, and a communication interface (I / F) 18. The camera apparatus 1 is implemented by an information processor such as a personal computer, for example.

[0030] FIG. 3 is a functional block diagram corresponding to the image processing apparatus 10. Such a functional configuration is implemented when hardware such as the CPU 12 and the DSP 13 reads and executes a program from the memory 14. In other words, the image processing apparatus 10 is obtained when the hardware such as the CPU 12 and the DSP 13 reads and executes a program from the memory 14. It is to be noted that the program does not need to be stored in the memory 14. For example, the program may be stored in a server. In such a case, the image processing apparatus 10 downloads the program from the server through a network and executes the program.

[0031] The image processing apparatus 10 includes an image data obtaining portion 100, a face image detecting portion 101, and an image organizing portion 102. The image data obtaining portion 100 receives image data from the camera 11 (S11).

[0032] The camera 11, similarly to a screen of a projector, is installed on an indoor wall or the like. The camera 11 is configured so as to capture a plurality of conference participants. Therefore, the camera 11 generates image data of an image that captures the plurality of conference participants. The camera 11 is an example of the image data obtaining portion. It is to be noted that the image processing apparatus 10 and the camera 11 do not need to be an integrally configured device as the camera apparatus 1 and may be respectively different devices. In such a case, the image processing apparatus 10 receives image data from the camera through the communication I / F such as a USB. In such a case, the communication I / F such as a USB configures the image data obtaining portion.

[0033] The face image detecting portion 101 detects a face image of each conference participant from received image data (S12). The method of detecting a face image may be any method. For example, the face image detecting portion 101 extracts an amount of characteristic of a face image from the received image data, and detects the face image by performing template matching with database of face images. The face image detecting portion 101 may detect the face image by a neural network. In addition, the face image detecting portion 101 may combine learning processes such as deep learning.

[0034] The information indicating a detection result of a face image is outputted to the image organizing portion 102. The image organizing portion 102 extracts the face image from the received image data, based on the detection result of a face image (S13).

[0035] FIG. 4 is a diagram showing an example of an image that the camera 11 has captured. The example of the image of FIG. 4 includes three conference participants. Therefore, the face image detecting portion 101 detects a face image of each of the three conference participants. The image organizing portion 102 extracts the face image of the three conference participants who have been detected. The image organizing portion 102 enlarges each of the extracted face images (hereinafter referred to as an individual face image) and reorganizes the face images into one image (S14).

[0036] The image organizing portion 102 outputs the reorganized image data through the communication I / F 18 (S15). For example, the reorganized image data is outputted, for example, to a display such as a projector or a monitor.

[0037] Accordingly, as shown in FIG. 5, the face images of all the conference participants are displayed as one image on the screen of a projector, or a monitor. Therefore, a conference participant can easily read each facial expression and can also easily read feeling.

[0038] It is to be noted that, in a case in which the conference is a teleconference, the image organizing portion 102 transmits an individual face image to a different device as packet data through the communication I / F 18. In such a case, the communication I / F 18 configures a packet transmitting portion. In addition, the image organizing portion 102 receives as packet data an individual face image outputted from the different device through the communication I / F 18. In such a case, the communication I / F 18 configures a packet receiving portion. The image organizing portion 102 enlarges and combines each of the individual face image extracted by the own device and the individual face image received from the different device, and reorganizes the face images into one image.

[0039] FIG. 6 is a diagram showing a flow of processing of each of a plurality of image processing apparatuses 10 that configure a teleconference. FIG. 7 is a flow chart showing an operation of the image processing apparatus 10. In FIG. 7, the same reference numerals are used to refer to processing common to the flow chart in FIG. 2, and the description is omitted.

[0040] In the plurality of image processing apparatuses 10, each image organizing portion 102 extracts a face image from an image captured by each camera 11 (S13). The each image organizing portion 102 transmits and receives an individual face image through the communication I / F 18 (S21). The image organizing portion 102 enlarges and combines each of the individual face image extracted by the own device and the individual face image received from the different device, and reorganizes the face images into one image (S22). The reorganized image data is outputted, for example, to a display such as a projector or a monitor (S23).

[0041] In this manner, the face images of all conference participants at the teleconference are displayed as one image on a screen or a monitor in each of respective locations that configure the teleconference. Therefore, a conference participant can easily read each facial expression and can also easily read feeling. It is to be noted that it is not always necessary to enlarge an individual face image. When all the conference participants are displayed in one image, a conference participant can see the faces of all the conference participants only by looking at one screen. Therefore, the conference participant can more easily read each facial expression and feeling than before.

[0042] It is to be noted that, in the processing of S14 in FIG. 2 or S22 in FIG. 6, the image organizing portion 102 may display content (a shared document, for example) related to the conference, together with the individual face image, in the image. For example, as shown in FIG. 8, some devices among the image processing apparatuses that configure the teleconference transmit a shared document. The image organizing portion 102, in the processing of S14 or S22, reorganizes the image including shared documents that have been transmitted and received.

[0043] As a result, the conference participants, since being able to converse while mutually looking at facial expressions and looking at the shared documents at the same time, can have a smooth conference.

[0044] It is to be noted that a reorganized image may be shared in all the image processing apparatuses 10 that configure the teleconference or the reorganized image of the conference participant on a near end side may be enlarged. However, when the same image is displayed in all the locations, the conference participants can easily recognize that all the conference participants are present in the same place and having a conference, without being conscious of a far end side or a near end side.

[0045] In addition, respective images of the conference participants may also be displayed in the same size, or the image of the conference participant on the near end side may be displayed large and the image of the conference participant on the far end side may be displayed small. In addition, a speaker may be detected and the individual face image of the speaker may be displayed large.

[0046] In addition, image processing does not need to be performed in a device in each location, and may be performed by a server. The device in each location transmits image data of an image captured by the camera 11, to the server. The server extracts an individual face image and reorganizes individual face images into one image. The server distributes reorganized image data to each location. In addition, the device in each location may extract an individual face image, for example. The server receives the individual face image from the device in each location and reorganizes individual face images into one image.

[0047] FIG. 9 is a functional block diagram of the image processing apparatus 10 further including a facial expression estimating portion 103, a feeling estimating portion 104, and a display mode changing portion 105. FIG. 10 is a flow chart showing an operation of the image processing apparatus 10. In FIG. 10, the same reference numerals are used to refer to processing common to the flow chart in FIG. 2, and the description is omitted.

[0048] Each configuration shown in FIG. 9 is also implemented when hardware such as the CPU 12 and the DSP 13 reads and executes a program from the memory 14.

[0049] The facial expression estimating portion 103 receives an individual face image from the image organizing portion 102 (S31). The facial expression estimating portion 103 estimates facial expression of each conference participant, based on the individual face image (S32). The method of estimating facial expression may be any method. For example, the facial expression estimating portion 103 extracts an amount of characteristic of a face image from received image data, and estimates facial expression by performing template matching with database of face images corresponding to various facial expressions. The facial expressions include, for example, laughter, surprise, cry, anger, darkness, brightness, or tension. In addition, the facial expressions also include blankness. The facial expression estimating portion 103 may estimates the facial expression by a neural network. In addition, the facial expression estimating portion 103 may combine learning processes such as deep learning.

[0050] The information indicating an estimated facial expression is outputted to the feeling estimating portion 104. The feeling estimating portion 104 estimates feeling of each conference participant, based on the facial expression estimated by the facial expression estimating portion 103 (S33). The method of estimating feeling may also be any method. For example, the feeling estimating portion 104 selects feeling matched with a received facial expression with reference to feeling database in which facial expression and feeling are associated with each other. The feeling, for example, includes various types of feelings such as fun, seriousness, boredom, impression, excitement, sadness, anger, abandonment, surprise, disgust, fear, embarrassment, or satisfaction. For example, in a case of the fear or the surprise, the facial expression corresponds to the feeling. In addition, the facial expression of laughter or bright facial expression highly correlates to the feeling of joy. In addition, the feeling estimating portion 104 may estimate feeling based on information related to a sight line. The feeling estimating portion 104 receives an individual face image from the image organizing portion 102, and extracts sight line information. It can be estimated that, when the sight line is directed to the front, seriousness may be high, and, when the sight line is directed in other directions, the feeling of idleness may be shown.

[0051] It is to be noted that it is not always necessary to estimate the feeling, based on a facial expression estimation result by the facial expression estimating portion 103. For example, the feeling estimating portion 104 extracts an amount of characteristic of a face image from received image data, and may directly estimate feeling by performing template matching with database of face images corresponding to various facial expressions. However, facial expression is an index for estimating a feeling. Therefore, the feeling estimating portion 104 is able to estimate feeling with higher accuracy by estimating the feeling based on the facial expression estimation result by the facial expression estimating portion 103.

[0052] In addition, the feeling estimating portion 104 may estimate the feeling by a neural network or may combine learning processes such as deep learning.

[0053] An estimation result of the feeling estimating portion 104 is outputted to the display mode changing portion 105. The display mode changing portion 105 receives image data from the image organizing portion 102 (S34). The display mode changing portion 105 changes a display mode of the face image of each conference participant, based on the feeling estimated by the feeling estimating portion 104 (S35). The image organizing portion 102 outputs the image data after the display mode is changed (S36). A change in the display mode includes a change in color, a change in position, a change in size, or animation addition.

[0054] For example, in the example of FIG. 11, the display mode changing portion 105 adds shaking animation to an image of a conference participant with a pleasant feeling. It is to be noted that the strength of animation may be changed, depending on the degree of feeling.

[0055] In addition, the display mode changing portion 105 may enlarge the image of a conference participant with a serious feeling, such as a speaker, and may reduce the image of a conference participant with a boring feeling. In addition, the display mode changing portion 105 may move upward the image of the conference participant with a serious feeling, such as a speaker, and may move downward the image of the conference participant with a bored feeling.

[0056] In addition, the display mode changing portion 105 may bring the images of conference participants with the similar feeling closer to each other. For example, the display mode changing portion 105 makes the images of the conference participants with a serious feeling adjacent to each other and also makes the images of the conference participants with a boring feeling adjacent to each other.

[0057] Alternatively, the display mode changing portion 105 may brightly display the images of a conference participant with a pleasant feeling and a conference participant with a serious feeling, and may darkly display the image of a conference participant with a bored feeling. In other words, the display mode changing portion 105 may change a color according to the feeling.

[0058] In the example shown from FIG. 9 to FIG. 11, reorganization of the individual face images by the image organizing portion 102 may not be performed in a background example. In such a background example, the image organizing portion 102 may output an image that captures a plurality of conference participants as it is, and the display mode changing portion 105 may only change a display mode of an image of each conference participant in the image that captures the plurality of conference participants. In this manner, even when reorganization of the individual face images is not performed, the conference participant can intuitively read the feeling of each conference participant from the image that displays a plurality of conference participants.

[0059] Finally, the present preferred embodiment is illustrative in all points and should not be construed to limit the present invention. The scope of the present invention is defined not by the foregoing preferred embodiment but by the following claims. Further, the scope of the present invention is intended to include all modifications within the scope of the appended claims.

Claims

1. An image processing apparatus comprising: a communication interface, I / F, (18); an image data obtaining portion (100) configured to obtain image data of an image that captures a plurality of conference participants at a first location of a teleconference; a face image detecting portion (101) configured to detect a face image of each of the plurality of conference participants from the image data obtained by the image data obtaining portion (100); an image organizing portion (102) configured to: extract each of the detected face images, transmit the detected face images to a different device at another location of the teleconference as packet data through the communication I / F (18), receive at least one other detected face image from the different device at the other location of the teleconference through the communication I / F (18), each of the at least one other detected face image corresponding to a conference participant at the other location of the teleconference, and reorganize both the extracted and received face images into one image; a facial expression estimating portion (103) configured to estimate facial expression of each conference participant based on the face image of said conference participant; a feeling estimating portion (104) configured to estimate a feeling of each conference participant based on the facial expression of said conference participant estimated by the facial expression estimating portion (103); and a display mode changing portion (105) configured to change a display mode of the face image of each conference participant based on the estimated feeling by changing a display position, a size or a color of the face image in the one image, or by adding animation to the face image, wherein the image organizing portion (102) is further configured to output, after the display mode is changed, the one image through the communication I / F (18) to a display in the first location of the teleconference.

2. The image processing apparatus according to claim 1, wherein the image organizing portion (102) is configured to display, in the one image, content related to a conference, together with the face image.

3. The image processing apparatus according to claim 1 or claim 2, wherein the image organizing portion (102) is configured to reorganize the face images of all the conference participants, in a same size, into the one image.

4. A camera apparatus comprising: the image processing apparatus according to any one of claims 1 to 3; and a camera to obtain the image data.

5. An image processing method comprising: obtaining image data of an image that captures a plurality of conference participants at a first location of a teleconference; detecting a face image of each of the plurality of conference participants from obtained image data; extracting each of the detected face images; transmitting the detected face images to a different device at another location of the teleconference as packet data through a communication interface, I / F, (18); receiving at least one other detected face image from the different device at the other location of the teleconference through the communication I / F (18), each of the at least one other detected face image corresponding to a conference participant at the other location of the teleconference; reorganizing both the extracted and received face images into one image; estimating facial expression of each conference participant based on the face image of said conference participant; estimating a feeling of each conference participant based on the estimated facial expression of said conference participant; changing a display mode of the face image of each conference participant based on the estimated feeling by changing a display position, a size or a color of the face image in the one image, or by adding animation to the face image; and outputting, after the display mode is changed, the one image through the communication I / F (18) to a display in the first location of the teleconference.

6. The image processing method according to claim 5, further comprising displaying in the one image, content related to a conference, together with the face image.