Image processing apparatus, program, and image processing method

Abstract

Provided are an image processing apparatus, a method, and a program for generating a face avatar that causes no discomfort regardless of whether an object exists in front of the face. 【Solution means】In an avatar display system, an image processing apparatus includes: an acquisition unit that acquires, from a communication device via a network, an RGB image generated by imaging a user with an imaging camera and a depth information image of the user having a lower resolution than the RGB image imaged by a depth camera; a noise removal image generation unit that generates a noise removal depth information image obtained by removing a background portion from the depth information image; a superimposed image generation unit that calibrates so as to match a user portion of the RGB image and a user portion of the noise removal depth information image, and superimposes the RGB image and the noise removal depth information image to generate a superimposed image; and an avatar generation unit that generates a face avatar 342 of the user based on the superimposed image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a program, and an image processing method.

Background Art

[0002] Patent Document 1 describes a technique for generating an avatar of a person based on the depth data and image data of the person. [Prior Art Document] [Patent Document] [Patent Document 1] Japanese Patent Application Laid-Open No. 2023-094549

Summary of the Invention

Means for Solving the Problems

[0003] According to one embodiment of the present invention, an image processing apparatus is provided. The image processing apparatus may include an acquisition unit that acquires an RGB image generated by imaging a user and a depth information image of the user having a lower resolution than the RGB image. The image processing apparatus may include a noise removal image generation unit that generates a noise removal depth information image by removing a background portion from the depth information image. The image processing apparatus may include a superimposed image generation unit that calibrates so that a user portion of the RGB image and a user portion of the noise removal depth information image coincide with each other, and superimposes the RGB image and the noise removal depth image to generate a superimposed image. The image processing apparatus may include an avatar generation unit that generates an avatar of the user's face based on the superimposed image.

[0004] In the image processing apparatus, the superimposed image generation unit may calibrate so that a user portion of the RGB image and a user portion of the noise removal depth information image coincide with each other according to a difference in resolution between the RGB image and the depth information image.

[0005] In any of the above image processing apparatuses, the superimposed image generation unit may adjust at least one of the size of the noise-removed depth information image and the positional relationship between the RGB image and the noise-removed depth information image so that the degree of coincidence between the user's part of the RGB image and the user's part of the noise-removed depth information image becomes higher, and superimpose the RGB image and the noise-removed depth information image to generate the superimposed image.

[0006] Any of the above image processing apparatuses may include a determination unit that determines, within a region corresponding to the face of the user, a portion where an object exists in front of the face of the user based on the superimposed image. Any of the above image processing apparatuses is a display control unit that controls to arrange and display the avatar on the face portion of the user in the RGB image, and arranges the corresponding portion of the RGB image in a portion where an object exists in front of the face of the user among the regions corresponding to the face of the user, and arranges the corresponding portion of the avatar in a region where no object exists in front of the face of the user, and may include a display control unit that controls to display a display image. The display control unit may control to display the display image in which the hand portion of the user in the RGB image is arranged in a portion where the hand of the user is located in front of the face of the user among the regions corresponding to the face of the user. The determination unit may calculate the average depth of the entire face of the user based on the superimposed image, and determine a portion where an object exists in front of the face of the user based on the average depth. The display control unit may determine, as a portion where an object exists in front of the face of the user, a portion where the depth is shallower than a predetermined threshold value by more than the average depth among the regions of the face of the user.

[0007] In any of the above image processing apparatuses, when the display control unit determines that the user within the imaging range cannot be detected while controlling to arrange and display the avatar on the face portion of the user in the RGB image of the user, the display control unit may control to fix and display the avatar that was being displayed immediately before the determination.

[0008] According to one embodiment of the present invention, a program for causing a computer to function as the image processing apparatus is provided.

[0009] According to one embodiment of the present invention, an image processing method executed by a computer is provided. The image processing method may include an acquisition step of acquiring an RGB image generated by imaging a user and a depth information image of the user having a lower resolution than the RGB image. The image processing method may include a noise removal image generation step of generating a noise removal depth information image in which a background portion is removed from the depth information image. The image processing method may include a superimposed image generation step of calibrating so that a user portion of the RGB image matches a user portion of the noise removal depth information image, and superimposing the RGB image and the noise removal depth information image to generate a superimposed image. The image processing method may include an avatar generation step of generating an avatar of the face of the user based on the superimposed image.

[0010] Note that the above summary of the invention does not list all the necessary features of the present invention. Also, sub-combinations of these feature groups can also be inventions.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0012] Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution means of the invention.

[0013] Techniques for generating a person's avatar using the person's depth data and image data are known. Depending on the apparatus that executes the avatar generation function, the performance of a depth camera that captures a person to generate the person's depth data may be lower than that of an imaging camera that captures a person to generate image data. The image processing apparatus 100 according to the present embodiment has a function of appropriately generating an avatar even when the performance of the depth camera is lower than that of the imaging camera.

[0014] FIG. 1 schematically shows an example of an avatar display system 10. In the example shown in FIG. 1, the avatar display system 10 includes an image processing apparatus 100. The image processing apparatus 100 includes an imaging camera 102 and a depth camera 104. The image processing apparatus 100 can be, for example, a smartphone, a tablet terminal, a PC (Personal Computer), etc., but is not limited thereto.

[0015] The imaging camera 102 provides a captured image to the image processing apparatus 100. The captured image may be an RGB image. The captured image may be a moving image. The captured image may be a still image captured continuously. The imaging camera 102 may be built into the image processing apparatus 100. The imaging camera 102 may be a camera externally attached to the image processing apparatus 100.

[0016] The depth camera 104 is a camera capable of generating a depth information image including depth information which is the distance to the imaging target. The depth information image includes depth information for each pixel. The depth camera 104 may also be called a depth camera. As the distance measurement method, known techniques can be used, for example, the triangulation method and the TOF (Time Of Flight) method, etc. The depth camera 104 may be a stereo camera. The depth camera 104 may be built into the image processing apparatus 100. The depth camera 104 may also be a camera externally attached to the image processing apparatus 100.

[0017] In the present embodiment, it is assumed that the resolution of the depth information image generated by the depth camera 104 is lower than that of the captured image generated by the imaging camera 102.

[0018] The image processing apparatus 100 acquires the RGB image generated by the imaging camera 102 capturing the user 40 from the imaging camera 102. The image processing apparatus 100 acquires the depth information image generated by the depth camera 104 capturing the user 40 from the depth camera 104. The depth information image generated by capturing the user 40 may be an image representing the depth (three-dimensional shape) of the user 40 as discrete point cloud data.

[0019] The image processing apparatus 100 may generate a noise-removed depth information image with the background part removed from the depth information image, calibrate so that the part of the user 40 in the RGB image and the part of the user 40 in the noise-removed depth information image match, and generate a superimposed image by superimposing the RGB image and the noise-removed depth information image.

[0020] The image processing apparatus 100 may generate an avatar of the face of the user 40 based on the superimposed image. The image processing apparatus 100 may determine a portion where an object exists in front of the face of the user 40 within a region corresponding to the face of the user 40 based on the superimposed image. The image processing apparatus 100 may control to display the corresponding portion of the RGB image in a region where an object exists in front of the face of the user 40 among the regions corresponding to the face of the user 40 in the RGB image, and display the corresponding portion of the avatar in a region where no object exists in front of the face of the user 40.

[0021] The image processing apparatus 100 may control to display the RGB image and the avatar on a display included in the image processing apparatus 100. The image processing apparatus 100 may also transmit display data for displaying the RGB image and the avatar to another apparatus, and control to display the RGB image and the avatar on the display of the other apparatus.

[0022] FIG. 2 schematically shows another example of the avatar display system 10. In the example shown in FIG. 2, the avatar display system 10 includes an image processing apparatus 100 and a communication apparatus 200. The communication apparatus 200 includes an imaging camera 202 and a depth camera 204. The communication apparatus 200 may be a smartphone, a tablet terminal, a PC, or the like, but is not limited thereto.

[0023] The imaging camera 202 may be the same as the imaging camera 102. The imaging camera 202 provides a captured image to the communication apparatus 200. The imaging camera 202 may be built in the communication apparatus 200. The imaging camera 202 may also be a camera externally attached to the communication apparatus 200.

[0024] The depth camera 204 may be the same as the depth camera 104. The imaging camera 202 is a camera capable of generating a depth information image including depth information which is the distance to the imaging target. The depth camera 204 may be built in the communication apparatus 200. The depth camera 204 may also be a camera externally attached to the communication apparatus 200.

[0025] In this embodiment, it is assumed that the resolution of the depth information image generated by the depth camera 204 is lower than that of the captured image generated by the imaging camera 202.

[0026] The image processing apparatus 100 receives, via the network 20, the RGB image generated by the imaging camera 202 capturing the user 40 from the communication apparatus 200. The image processing apparatus 100 receives, via the network 20, the depth information image generated by the depth camera 204 capturing the user 40 from the communication apparatus 200.

[0027] The network 20 may include the Internet. The network 20 may include a LAN (Local Area Network). The network 20 may include a mobile communication network. The mobile communication network may conform to any of the communication systems such as the LTE (Long Term Evolution) communication system, 5G (5th Generation) communication system, 3G (3rd Generation) communication system, and communication systems after 6G (6th Generation).

[0028] The image processing apparatus 100 may be connected to the network 20 by wire. The image processing apparatus 100 may also be wirelessly connected to the network 20. The image processing apparatus 100 may be connected to the network 20 via a wireless base station. The image processing apparatus 100 may be connected to the network 20 via a Wi-Fi (registered trademark) access point.

[0029] The communication apparatus 200 may be connected to the network 20 by wire. The communication apparatus 200 may also be wirelessly connected to the network 20. The communication apparatus 200 may be connected to the network 20 via a wireless base station. The communication apparatus 200 may be connected to the network 20 via a Wi-Fi access point.

[0030] The image processing apparatus 100 may generate a noise-removed depth information image with the background portion removed from the depth information image, calibrate so that the portion of the user 40 in the RGB image matches the portion of the user 40 in the noise-removed depth information image, and generate a superimposed image by superimposing the RGB image and the noise-removed depth information image.

[0031] The image processing apparatus 100 may generate an avatar of the face of the user 40 based on the superimposed image. The image processing apparatus 100 may determine, within the region corresponding to the face of the user 40 based on the superimposed image, a portion where an object exists in front of the face of the user 40. The image processing apparatus 100 may display the corresponding portion of the RGB image in the region corresponding to the face of the user 40 in the RGB image where an object exists in front of the face of the user 40, and control to display the corresponding portion of the avatar in the region where no object exists in front of the face of the user 40.

[0032] The image processing apparatus 100 may control to display the RGB image and the avatar on a display included in the image processing apparatus 100. The image processing apparatus 100 may also transmit display data for displaying the RGB image and the avatar to the communication apparatus 200 and control to display the RGB image and the avatar on the display of the communication apparatus 200. The image processing apparatus 100 may also transmit display data for displaying the RGB image and the avatar to a device other than the communication apparatus 200 and control to display the RGB image and the avatar on the display of the said device.

[0033] FIG. 3 is an explanatory diagram for schematically explaining avatar generation by the image processing apparatus 100. Here, the case where no object exists in front of the face of the user 40 will be described.

[0034] The image processing apparatus 100 generates a noise-removed depth information image 312 with the background portion removed from the depth information image 310. The image processing apparatus 100, for example, removes a portion deeper than a predetermined threshold depth from the depth information image 310. The image processing apparatus 100, for example, removes the background portion specified from the average depth of the depth information image 310.

[0035] The image processing apparatus 100 calibrates so as to match the portion of the user 40 in the RGB image 320 and the portion of the user 40 in the noise-removed depth information image 312, and superimposes the RGB image 320 and the noise-removed depth information image 312 to generate a superimposed image 330.

[0036] The image processing apparatus 100 generates an avatar of the face of the user 40 based on the superimposed image 330. The image processing apparatus 100 determines, within the region corresponding to the face of the user 40 based on the superimposed image 330, a portion where an object exists in front of the face of the user 40. In this example, it is determined that no object exists.

[0037] The image processing apparatus 100 may control to display a display image 340 in which a portion other than the face of the user 40 is a real image and the portion of the face of the user 40 is an avatar 342 that moves in accordance with the movement of the face of the user 40. The image processing apparatus 100 may control to display a display image 340 in which the corresponding portion of the RGB image 320 is arranged in a portion other than the region corresponding to the face of the user 40 in the RGB image 320, and the generated avatar 342 is arranged in the portion of the face of the user 40 in the RGB image 320. By continuously executing such processing, it is possible to display a display image 340 including the avatar 342 that follows the movement of the user 40.

[0038] FIG. 4 is an explanatory diagram for schematically explaining avatar generation by the image processing apparatus 100. Here, a case where the hand of the user 40 exists in front of the face of the user 40 will be described. Up to the generation of the superimposed image 330, it is the same as in FIG. 3.

[0039] The image processing apparatus 100 generates an avatar of the face of the user 40 based on the superimposed image 330. The image processing apparatus 100 determines, within the region corresponding to the face of the user 40 based on the superimposed image 330, a portion where an object exists in front of the face of the user 40. In this example, it is determined that an object exists.

[0040] The image processing apparatus 100 may control to display a display image 340 in which corresponding portions of the RGB image 320 are arranged in portions other than the region corresponding to the face of the user 40 in the RGB image 320, corresponding portions of the RGB image 320 are arranged in portions of the region corresponding to the face of the user 40 in the RGB image 320 where an object exists in front of the face of the user 40, and corresponding portions of the avatar 342 are arranged in a region where no object exists in front of the face of the user 40. Thereby, when the hand of the user 40 exists in front of the face of the user 40, corresponding portions of the RGB image 320 are arranged in portions other than the region corresponding to the face of the user 40, and in the region corresponding to the face of the user 40, corresponding portions of the avatar 342 are arranged in portions where the hand does not exist, and corresponding portions of the RGB image 320 are arranged in portions where the hand exists, and the display image 340 can be displayed.

[0041] FIG. 5 shows an example of a portion 354 where a hand exists in a region 352 corresponding to the face of the user 40 in the display image 340. In FIG. 5, the region 352 is surrounded by a solid line, and the portion 354 is surrounded by a broken line. The image processing apparatus 100 may identify the region corresponding to the face of the user 40 by analyzing the superimposed image 330. The image processing apparatus 100 may determine whether each pixel in the region corresponding to the face of the user 40 is a face or a hand by referring to the depth.

[0042] The image processing apparatus 100, for example, calculates the average depth of the face of the user 40 in advance, and based on the average depth, determines whether each pixel in the region corresponding to the face of the user 40 is a face or a hand. For example, for each pixel, the image processing apparatus 100 determines that it is a hand when the depth is shallower than a predetermined threshold value with respect to the average depth, and determines that it is a face when it is not shallower. That is, the image processing apparatus 100 determines, as a portion where a hand exists, a portion of the region corresponding to the face of the user 40 where the depth is shallower than a predetermined threshold value with respect to the average depth.

[0043] The image processing apparatus 100 calculates the average depth of the face of the user 40, for example, from the superimposed image 330 in a state where no object exists in front of the face of the user 40. The image processing apparatus 100 analyzes the superimposed image 330 to identify the region corresponding to the face of the user 40, and calculates the average depth from the depths of the respective pixels in the identified region.

[0044] The image processing apparatus 100 may calculate the average depth of the face of the user 40 from the superimposed image 330 in a state where an object exists in front of the face of the user 40. The image processing apparatus 100 analyzes the superimposed image 330 to identify the region corresponding to the face of the user 40, and calculates the average depth from the depths of the respective pixels in the portion other than the portion with a high probability of the existence of an object in the identified region. The image processing apparatus 100, for example, divides the identified region into a group of pixels with a deep depth and a group of pixels with a shallow depth, and calculates the average depth from the depths of the respective pixels in the group of pixels with a deep depth.

[0045] The image processing apparatus 100 may use the average depth of the entire face of the user 40 as the average depth of the face of the user 40. The image processing apparatus 100 may also use the average depth of a predetermined region of the face of the user 40 as the average depth of the face of the user 40. The predetermined region may be arbitrarily set, for example, a region including the eyes, nose, and mouth of the user 40.

[0046] Note that the image processing apparatus 100 may determine whether each pixel in the region corresponding to the face of the user 40 is a face or a hand based on the depth of a single point on the face of the user 40, for example. As a specific example, the image processing apparatus 100 determines whether each pixel in the region corresponding to the face of the user 40 is a face or a hand based on the depth of the midpoint of the face of the user 40. The image processing apparatus 100 uses, for example, the apex of the nose of the user 40 as the midpoint of the face of the user 40. The image processing apparatus 100 may use the midpoint of the region corresponding to the face of the user 40 as the midpoint of the face of the user 40. For example, for each pixel, if the depth is shallower than the depth of the midpoint by a predetermined threshold or more, the image processing apparatus 100 determines that it is a hand, and if it is not shallower, the image processing apparatus 100 determines that it is a face. That is, the image processing apparatus 100 determines, as the portion where the hand exists, the portion in the region corresponding to the face of the user 40 where the depth is shallower than the depth of the midpoint by a predetermined threshold or more. Thereby, the processing load can be reduced. However, when the depth of the midpoint of the face of the user 40 is not accurate, there is a possibility that a determination error regarding whether an object exists in front of the face of the user 40, so-called occlusion error, may occur. On the other hand, as described above, the occlusion error can be reduced by using the average depth of the face of the user 40.

[0047] FIG. 6 schematically shows an example of the functional configuration of the image processing apparatus 100. The image processing apparatus 100 includes an acquisition unit 112, a superimposed image generation unit 114, a noise removal image generation unit 116, a determination unit 118, an avatar generation unit 120, and a display control unit 122. Note that it is not always essential for the image processing apparatus 100 to include all of these.

[0048] The acquisition unit 112 acquires the RGB image 320 of the user 40 generated by imaging the user 40. The acquisition unit 112 may acquire the RGB image 320 generated by the imaging camera 102 from the imaging camera 102. The acquisition unit 112 may receive the RGB image 320 generated by the imaging camera 202 from the communication device 200.

[0049] The acquisition unit 112 acquires the depth information image 310 of the user 40 generated by imaging the user 40. The acquisition unit 112 acquires the depth information image 310 with a lower resolution than the RGB image 320. The acquisition unit 112 may acquire the depth information image 310 generated by the depth camera 104 from the depth camera 104. The acquisition unit 112 may receive the depth information image 310 generated by the depth camera 204 from the communication device 200.

[0050] The noise removal image generation unit 116 generates a noise removal depth information image 312 from which the background portion has been removed from the depth information image 310. The noise removal image generation unit 116 removes, for example, portions deeper than a predetermined threshold depth from the depth information image 310. The noise removal image generation unit 116 removes, for example, the background portion specified from the average depth of the depth information image 310.

[0051] The superimposed image generation unit 114 calibrates so that the user portion of the RGB image 320 and the user portion of the noise removal depth information image 312 coincide, and superimposes the RGB image 320 and the noise removal depth information image 312 to generate a superimposed image 330.

[0052] The superimposed image generation unit 114 may calibrate so that the user portion of the RGB image 320 and the user portion of the noise removal depth information image 312 coincide according to the difference in resolution between the RGB image 320 and the depth information image 310. For example, the superimposed image generation unit 114 enlarges the noise removal depth information image 312 by a magnification according to the difference in resolution between the RGB image 320 and the depth information image 310. As another example, the superimposed image generation unit 114 may reduce the RGB image 320 by a magnification according to the difference in resolution between the RGB image 320 and the depth information image 310.

[0053] The superimposed image generation unit 114 may adjust at least one of the size of the noise-removed depth information image 312 and the positional relationship between the RGB image 320 and the noise-removed depth information image 312 so that the degree of coincidence between the user part of the RGB image 320 and the user part of the noise-removed depth information image 312 becomes higher, and superimpose the RGB image 320 and the noise-removed depth information image 312 to generate a superimposed image 330.

[0054] For example, the superimposed image generation unit 114 adjusts the size of the noise-removed depth information image 312 so that the degree of coincidence between the user part of the RGB image 320 and the user part of the noise-removed depth information image 312 becomes higher. As a specific example, the superimposed image generation unit 114 calculates the degree of coincidence between the user part of the RGB image 320 and the user part of the noise-removed depth information image 312 while gradually enlarging the size of the noise-removed depth information image 312, and superimposes the RGB image 320 and the noise-removed depth information image 312 with the size at which the degree of coincidence becomes the highest.

[0055] For example, the superimposed image generation unit 114 adjusts the positional relationship between the RGB image 320 and the noise-removed depth information image 312 so that the degree of coincidence between the user part of the RGB image 320 and the user part of the noise-removed depth information image 312 becomes higher. The superimposed image generation unit 114 adjusts, for example, the relative position between the RGB image 320 and the noise-removed depth information image 312. The superimposed image generation unit 114 adjusts, for example, the relative angle between the RGB image 320 and the noise-removed depth information image 312. The superimposed image generation unit 114 adjusts, for example, the relative position and the relative angle between the RGB image 320 and the noise-removed depth information image 312. As a specific example, the superimposed image generation unit 114 enlarges the noise-removed depth information image 312 by a magnification corresponding to the difference in resolution between the RGB image 320 and the depth information image 310, and then calculates the degree of coincidence between the user part of the RGB image 320 and the user part of the noise-removed depth information image 312 while gradually changing the positional relationship between the RGB image 320 and the noise-removed depth information image 312, and superimposes the RGB image 320 and the noise-removed depth information image 312 with the positional relationship at which the degree of coincidence becomes the highest.

[0056] For example, the superimposed image generation unit 114 adjusts both the size of the noise-removed depth information image 312 and the positional relationship between the RGB image 320 and the noise-removed depth information image 312 so that the degree of coincidence between the user part of the RGB image 320 and the user part of the noise-removed depth information image 312 becomes higher. As a specific example, the superimposed image generation unit 114 gradually calculates the degree of coincidence between the user part of the RGB image 320 and the user part of the noise-removed depth information image 312 while expanding the size of the noise-removed depth information image 312 step by step or changing the positional relationship between the RGB image 320 and the noise-removed depth information image 312, and superimposes the RGB image 320 and the noise-removed depth information image 312 with the size of the noise-removed depth information image 312 and the positional relationship between the RGB image 320 and the noise-removed depth information image 312 where the degree of coincidence is the highest.

[0057] Due to the positional relationship between the imaging camera 102 and the depth camera 104, when the noise-removed depth information image 312 is simply enlarged and superimposed on the RGB image 320, the user part in the RGB image 320 and the user part in the noise-removed depth information image 312 may not exactly match. In such a case, there is a possibility that the avatar 342 cannot be accurately generated or the object existing in front of the face of the user 40 cannot be accurately recognized. However, by performing the calibration described above by the superimposed image generation unit 114, such a possibility can be reduced.

[0058] The superimposed image generation unit 114 may perform calibration so as to match the user part of the RGB image 320 with the user part of the depth information image 310 acquired by the acquisition unit 112 instead of the noise-removed depth information image 312, and generate a superimposed image 330 by superimposing the RGB image 320 and the depth information image 310.

[0059] Based on the superimposed image 330, the determination unit 118 determines a portion where an object exists in front of the face of the user 40 within the region corresponding to the face of the user 40. The determination unit 118 may calculate the average depth of the user's face based on the superimposed image 330 and determine a portion where an object exists in front of the user's face based on the average depth.

[0060] For example, the determination unit 118 calculates the average depth of the face of the user 40 in advance, and based on the average depth, determines whether each pixel within the region corresponding to the face of the user 40 is a face or an object existing in front of the face. For example, for each pixel, when the depth is shallower than a predetermined threshold value from the average depth, the determination unit 118 determines that it is an object existing in front of the face, and when it is not shallower, determines that it is a face. That is, the determination unit 118 determines a portion where the depth is shallower than a predetermined threshold value from the average depth within the region corresponding to the face of the user 40 as a portion where an object exists.

[0061] For example, the determination unit 118 calculates the average depth of the face of the user 40 from the superimposed image 330 in a state where no object exists in front of the face of the user 40. The determination unit 118 identifies the region corresponding to the face of the user 40 by analyzing the superimposed image 330, and calculates the average depth from the depths of each pixel in the identified region.

[0062] The determination unit 118 may also calculate the average depth of the face of the user 40 from 300 in a state where an object exists in front of the face of the user 40. The determination unit 118 identifies the region corresponding to the face of the user 40 by analyzing the superimposed image 330, and calculates the average depth from the depths of each pixel in a portion other than a portion where the probability of an object existing is high within the identified region. The image processing apparatus 100, for example, divides the identified region into a group of pixels with a deep depth and a group of pixels with a shallow depth, and calculates the average depth from the depths of each pixel in the group of pixels with a deep depth.

[0063] The determination unit 118 may use the average depth of the entire face of the user 40 as the average depth of the face of the user 40. The determination unit 118 may also use the average depth of a predetermined region of the face of the user 40 as the average depth of the face of the user 40. The predetermined region may be arbitrarily set, for example, a region including the eyes, nose, and mouth of the user 40, etc.

[0064] In addition, the determination unit 118 may determine whether each pixel in the region corresponding to the face of the user 40 is a face or a hand based on, for example, the depth of a single point on the face of the user 40. As a specific example, the determination unit 118 determines whether each pixel in the region corresponding to the face of the user 40 is a face or a hand based on the depth of the midpoint of the face of the user 40. The determination unit 118, for example, uses the apex of the nose of the user 40 as the midpoint of the face of the user 40. The determination unit 118 may also use the midpoint of the region corresponding to the face of the user 40 as the midpoint of the face of the user 40.

[0065] The avatar generation unit 120 generates an avatar 342 of the face of the user 40 based on the superimposed image 330. For example, the avatar generation unit 120 uses the noise-removed depth information image 312 in the superimposed image 330 to generate a 3D model of the face of the user 40, uses the RGB image 320 in the superimposed image 330 to generate texture data of the face of the user 40, and performs a process of attaching the texture data to the 3D model to generate the avatar 342. The avatar generation unit 120 may generate the avatar 342 from the superimposed image 330 using other methods.

[0066] The display control unit 122 controls to display the display image 340 according to the determination result by the determination unit 118. When the determination unit 118 determines that there is no object in front of the face of the user 40, the display control unit 122 arranges the corresponding part of the RGB image 320 in the part other than the region corresponding to the face of the user 40 in the RGB image 320, and arranges the avatar 342 generated by the avatar generation unit 120 in the face part of the RGB image 320, and controls to display the display image 340.

[0067] When the determination unit 118 determines that an object exists in front of the face of the user 40, the display control unit 122 arranges the corresponding portion of the RGB image 320 in the portion other than the region corresponding to the face of the user 40 in the RGB image 320, and arranges the corresponding portion of the RGB image 320 in the portion of the region corresponding to the face of the user 40 in the RGB image 320 where the object exists in front of the face of the user 40, and arranges the corresponding portion of the avatar 342 in the region where no object exists in front of the face of the user 40, and may control to display the display image 340. For example, when the hand of the user 40 exists in front of the face of the user 40, the display control unit 122 controls to display the display image 340 in which the hand portion of the user 40 in the RGB image 320 is arranged in the portion of the region corresponding to the face of the user 40 where the hand of the user 40 exists in front of the face of the user 40.

[0068] When it is determined that the user 40 within the imaging range cannot be detected while the display control unit 122 is controlling to arrange and display the avatar 342 in the face portion of the user in the RGB image 320, the display control unit 122 may control to fix and display the displayed avatar 342. When the user 40 within the imaging range is missed, there is a possibility that a situation may occur where the avatar 342 disappears and the real image of the face of the user 40 is displayed, which is troublesome. However, as described above, by controlling the display control unit 122 to fix and display the avatar 342, the avatar 342 can be fixed at the final position, and it is possible to prevent the avatar 342 from disappearing and the face of the person from being displayed.

[0069] FIG. 7 schematically shows an example of the hardware configuration of a computer 1200 that functions as the image processing apparatus 100 or the communication apparatus 200. Programs installed in the computer 1200 can cause the computer 1200 to function as one or more “units” of the apparatus according to the present embodiment, or can cause the computer 1200 to execute operations associated with the apparatus according to the present embodiment or the one or more “units”, and / or can cause the computer 1200 to execute the process according to the present embodiment or a stage of the process. Such a program may be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0070] The computer 1200 according to the present embodiment includes a CPU 1212, a RAM 1214, and a graphic controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes legacy input / output units such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0071] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 in a frame buffer or the like provided in the RAM 1214 or in itself, and causes the image data to be displayed on the display device 1218.

[0072] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0073] The ROM 1230 stores therein a boot program or the like executed by the computer 1200 at activation and / or a program dependent on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, or the like.

[0074] The program is provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The program is read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which is also an example of a computer-readable storage medium, and executed by the CPU 1212. The information processing described in these programs is read by the computer 1200, resulting in cooperation between the programs and the various types of hardware resources described above. The device or method may be configured by realizing the operation or processing of information according to the use of the computer 1200.

[0075] For example, when communication is executed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in a recording medium such as the RAM 1214, the storage device 1224, the DVD-ROM, or the IC card, transmits the read transmission data to the network, or writes the received data received from the network to a reception buffer area or the like provided on the recording medium.

[0076] Further, the CPU 1212 may cause all or a necessary part of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive (DVD-ROM), the IC card, etc. to be read into the RAM 1214 and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0077] Various types of information such as various types of programs, data, tables, and databases may be stored in the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, search / replacement of information, etc. described throughout this disclosure and specified by the program instruction sequence, and write back the result to the RAM 1214. Further, the CPU 1212 may search for information in files, databases, etc. within the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 searches for an entry that matches the condition where the attribute value of the first attribute is specified among the plurality of entries, reads the attribute value of the second attribute stored in the entry, and thereby obtains the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0078] The programs or software modules described above may be stored on a computer-readable storage medium on or near the computer 1200. Also, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the program to the computer 1200 via the network.

[0079] The blocks in the flowcharts and block diagrams in this embodiment may represent stages of a process in which operations are performed or "parts" of an apparatus having a role of performing operations. Specific stages and "parts" may be implemented by a dedicated circuit, a programmable circuit supplied with computer-readable instructions stored on a computer-readable storage medium, and / or a processor supplied with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuit may include digital and / or analog hardware circuits and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include, for example, a reconfigurable hardware circuit including logical products, logical sums, exclusive logical sums, negative logical products, negative logical sums, and other logical operations, flip-flops, registers, and memory elements, such as a field programmable gate array (FPGA) and a programmable logic array (PLA).

[0080] A computer-readable storage medium may include any tangible device that can store instructions executable by a suitable device. As a result, a computer-readable storage medium having instructions stored therein will comprise a product that includes instructions that can be executed to create means for performing the operations specified in a flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy (registered trademark) disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD), Blu-ray (registered trademark) disk, memory stick, integrated circuit card, and the like.

[0081] Computer-readable instructions may include any combination of one or more programming languages, including source code or object code written in any combination of assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming languages such as Smalltalk (registered trademark), JAVA (registered trademark), C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages.

[0082] Computer-readable instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or a programmable circuit, locally or via a wide area network (WAN) such as a local area network (LAN), the Internet, etc., to execute the computer-readable instructions to generate means for performing the operations specified in a flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, and the like.

[0083] As described above, the present invention has been described using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that forms with such changes or improvements can also be included in the technical scope of the present invention.

[0084] It should be noted that the execution order of each process such as operations, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, the specification, and the drawings is not explicitly indicated as "earlier" or "preceding" etc., and can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flowcharts in the claims, the specification, and the drawings, even if "first," "next," etc. are used for convenience of explanation, it does not mean that it is essential to implement in this order.

Description of Reference Numerals

[0085] 10 Avatar display system, 20 Network, 40 User, 100 Image processing device, 102 Imaging camera, 104 Depth camera, 112 Acquisition unit, 114 Superimposed image generation unit, 116 Noise removal image generation unit, 118 Determination unit, 120 Avatar generation unit, 122 Display control unit, 200 Communication device, 202 Imaging camera, 204 Depth camera, 310 Depth information image, 312 Noise removal depth information image, 320 RGB image, 330 Superimposed image, 340 Display image, 342 Avatar, 352 Region, 354 Portion, 1200 Computer, 1210 Host controller, 1212 CPU, 1214 RAM, 1216 Graphics controller, 1218 Display device, 1220 Input / output controller, 1222 Communication interface, 1224 Storage device, 1230 ROM, 1240 Input / output chip

Claims

1. An acquisition unit that acquires an RGB image generated by imaging a user and a depth information image of the user that has a lower resolution than the RGB image; A noise removal image generation unit that generates a noise removal depth information image by removing a background portion from the depth information image; A superimposed image generation unit that calibrates so that a user portion of the RGB image and a user portion of the noise removal depth information image match, and superimposes the RGB image and the noise removal depth information image to generate a superimposed image; An avatar generation unit that generates an avatar of the user's face based on the superimposed image; Based on the superimposed image, calculates an average depth of a region including the eyes, nose, and mouth of the user's face, and determines, among regions corresponding to the user's face, a portion having a depth shallower than the average depth by a predetermined threshold or more as a portion where an object exists in front of the user's face; a determination unit; A display control unit that controls to arrange and display the avatar on a face portion of the user in the RGB image, and arranges a corresponding portion of the RGB image in a portion where an object exists in front of the user's face among regions corresponding to the user's face, and arranges a corresponding portion of the avatar in a region where no object exists in front of the user's face, and controls to display a display image; a display control unit; An image processing apparatus comprising the above.

2. The superimposed image generation unit calibrates so that a user portion of the RGB image and a user portion of the noise removal depth information image match according to a difference in resolution between the RGB image and the depth information image. The image processing apparatus according to claim 1.

3. The superimposed image generation unit enlarges the noise removal depth information image at a magnification according to a difference in resolution between the RGB image and the depth information image. The image processing apparatus according to claim 2.

4. The superimposed image generation unit adjusts at least one of the size of the noise removal depth information image and the positional relationship between the RGB image and the noise removal depth information image so that the degree of coincidence between a user portion of the RGB image and a user portion of the noise removal depth information image becomes higher, and superimposes the RGB image and the noise removal depth information image to generate the superimposed image. The image processing apparatus according to claim 1.

5. The image processing device described in Claim 4, wherein the superimposed image generation unit calculates the degree of correspondence between the user's part of the RGB image and the user's part of the noise-removed depth information image while gradually increasing the size of the noise-removed depth information image, and superimposes the RGB image and the noise-removed depth information image at the size at which the degree of correspondence is highest.

6. The image processing device described in claim 4, wherein the superimposed image generation unit enlarges the noise-removed depth information image by a magnification corresponding to the difference in resolution between the RGB image and the depth information image, and then calculates the degree of correspondence between the user's part of the RGB image and the user's part of the noise-removed depth information image while gradually changing the positional relationship between the RGB image and the noise-removed depth information image, and superimposes the RGB image and the noise-removed depth information image at the positional relationship that results in the highest degree of correspondence.

7. The image processing device described in claim 4, wherein the superimposed image generation unit calculates the degree of match between the user's part of the RGB image and the user's part of the noise-removed depth information image while gradually increasing the size of the noise-removed depth information image or changing the positional relationship between the RGB image and the noise-removed depth information image, and superimposes the RGB image and the noise-removed depth information image at the size of the noise-removed depth information image and the positional relationship between the RGB image and the noise-removed depth information image that results in the highest degree of match.

8. 8. The image processing device according to claim 1, wherein the display control unit controls the display image to display a portion of the user's hand in the RGB image in a region corresponding to the user's face where the user's hand is located in front of the user's face.

9. 8. The image processing device according to claim 1, wherein the display control unit controls the avatar to be displayed in a fixed position when it determines that the user cannot be detected within the imaging range while controlling the avatar to be displayed in the area of the user's face in the RGB image.

10. A program for causing a computer to function as the image processing device according to any one of claims 1 to 7.

11. 1. A computer-implemented image processing method comprising: An acquisition step of acquiring an RGB image generated by imaging a user and a depth information image of the user having a lower resolution than the RGB image; A noise removal image generation step of generating a noise removal depth information image obtained by removing a background portion from the depth information image; A superimposed image generation step of calibrating so that a user portion of the RGB image and a user portion of the noise removal depth information image coincide with each other, and superimposing the RGB image and the noise removal depth information image to generate a superimposed image; An avatar generation step of generating an avatar of the user's face based on the superimposed image; A determination step of calculating an average depth of a region including eyes, nose, and mouth of the user's face based on the superimposed image, and determining, among regions corresponding to the user's face, a portion having a depth shallower than the average depth by a predetermined threshold or more as a portion where an object exists in front of the user's face; A display control step of controlling to arrange and display the avatar on a face portion of the user in the RGB image, arranging a corresponding portion of the RGB image in a portion where an object exists in front of the user's face among regions corresponding to the user's face, and arranging a corresponding portion of the avatar in a region where no object exists in front of the user's face, and controlling to display a display image; An image processing method comprising the above steps.

Images