Image processing device, program, and image processing method

Abstract

To provide an image processing device capable of generating a natural-looking facial avatar regardless of whether an object is present in front of the face, a program, and an information processing method.SOLUTION: In an avatar display system, an image processing device comprises: an acquisition unit which acquires an RGB image generated by capturing a user by an imaging camera and a depth information image of the user having a lower resolution than the RGB image captured by a depth camera via a network from a communication device; a noise removal image generation unit which generates a noise-removed depth information image where a background portion is removed from the depth information image; a superimposed image generation unit which generates a superimposed image by superimposing the RGB image on the noise-removed depth information image by performing calibration for aligning the user's portion of the RGB image with the user's portion of the noise-removed depth information image; and an avatar generation unit which generates a facial avatar 342 of the user on the basis of the superimposed image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a program, and an image processing method. [Background technology]

[0002] Patent Document 1 describes a technique for generating an avatar of a person based on depth data and image data of the person. [Prior art document] [Patent documents] [Patent Document 1] JP 2023-094549 A Summary of the Invention [Means for solving the problem]

[0003] According to one embodiment of the present invention, there is provided an image processing device. The image processing device may include an acquisition unit that acquires an RGB image and a depth information image of the user having a lower resolution than the RGB image, both of which are generated by capturing an image of the user. The image processing device may include a noise-removed image generation unit that generates a noise-removed depth information image by removing a background portion from the depth information image. The image processing device may include a superimposed image generation unit that performs calibration to match a portion of the user in the RGB image with a portion of the user in the noise-removed depth information image, and superimposes the RGB image and the noise-removed depth image to generate a superimposed image. The image processing device may include an avatar generation unit that generates an avatar of the user's face based on the superimposed image.

[0004] In the image processing device, the superimposed image generation unit may perform calibration so as to match the user's portion in the RGB image with the user's portion in the noise-removed depth information image, depending on the difference in resolution between the RGB image and the depth information image.

[0005] In any of the image processing devices, the superimposed image generation unit may adjust at least one of the size of the noise-removed depth information image and the positional relationship between the RGB image and the noise-removed depth information image so as to increase the degree of correspondence between the user's portion in the RGB image and the user's portion in the noise-removed depth information image, and then superimpose the RGB image and the noise-removed depth information image to generate the superimposed image.

[0006] Any of the image processing devices may include a determination unit that determines, based on the superimposed image, a portion of a region corresponding to the user's face where an object exists in front of the user's face. Any of the image processing devices may include a display control unit that controls the avatar to be placed and displayed in the region of the user's face in the RGB image, and that controls the display control unit to place the corresponding portion of the RGB image in a region of the region corresponding to the user's face where an object exists in front of the user's face, and to display a display image in which the corresponding portion of the avatar is placed in a region of the region corresponding to the user's face where no object exists in front of the user's face. The display control unit may control the display image to be displayed in which the user's hand in the RGB image is placed in a region of the region corresponding to the user's face where the user's hand is located in front of the user's face. The determination unit may calculate an average depth of the entire user's face based on the superimposed image, and determine, based on the average depth, a portion of the region corresponding to the user's face where an object exists in front of the user's face. The display control unit may determine that a portion of the user's face area that is shallower than the average depth by a predetermined threshold or more is a portion where an object exists in front of the user's face.

[0007] In any of the image processing devices, when the display control unit is controlling the display of the avatar by placing it in the area of ​​the user's face in the RGB image of the user, and it determines that the user cannot be detected within the imaging range, it may control the display of the avatar that was displayed immediately before making the determination to be fixed.

[0008] According to one embodiment of the present invention, there is provided a program for causing a computer to function as the image processing device.

[0009] According to one embodiment of the present invention, there is provided an image processing method executed by a computer. The image processing method may include an acquisition step of acquiring an RGB image and a depth information image of the user having a lower resolution than the RGB image, both of which are generated by capturing an image of the user. The image processing method may include a noise-removed image generation step of generating a noise-removed depth information image by removing a background portion from the depth information image. The image processing method may include a superimposed image generation step of performing calibration so that a portion of the user in the RGB image matches a portion of the user in the noise-removed depth information image, and superimposing the RGB image and the noise-removed depth information image to generate a superimposed image. The image processing method may include an avatar generation step of generating an avatar of the user's face based on the superimposed image.

[0010] The above summary of the invention does not list all of the necessary features of the present invention, and subcombinations of these features may also constitute inventions. [Brief explanation of the drawings]

[0011] [Figure 1] 1 illustrates a schematic diagram of an example avatar display system 10. [Figure 2] 1 shows another example of an avatar display system 10 in a simplified manner. [Figure 3] FIG. 2 is an explanatory diagram for schematically explaining avatar generation by the image processing device 100. [Figure 4] FIG. 2 is an explanatory diagram for schematically explaining avatar generation by the image processing device 100. [Figure 5] An example of a portion of the area corresponding to the face of the user 40 where the hands are present and where the hands are not present is shown. [Figure 6] 1 shows an example of a functional configuration of an image processing device 100. [Figure 7]1 shows an example of a hardware configuration of a computer 1200 that functions as the image processing device 100 or the communication device 200. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0013] There is known a technology for generating a person's avatar using depth data and image data of the person. Depending on the device that executes the avatar generation function, the performance of the depth camera that captures the person and generates the person's depth data may be lower than that of the imaging camera that captures the person and generates image data. The image processing device 100 according to this embodiment has a function for appropriately generating an avatar even when the performance of the depth camera is lower than that of the imaging camera.

[0014] Fig. 1 schematically illustrates an example of an avatar display system 10. In the example illustrated in Fig. 1, the avatar display system 10 includes an image processing device 100. The image processing device 100 includes an imaging camera 102 and a depth camera 104. The image processing device 100 may be, but is not limited to, a smartphone, a tablet terminal, a PC (Personal Computer), or the like.

[0015] The imaging camera 102 provides the captured image to the image processing device 100. The captured image may be an RGB image. The captured image may be a moving image. The captured image may be continuously captured still images. The imaging camera 102 may be built into the image processing device 100. The imaging camera 102 may be an external camera attached to the image processing device 100.

[0016] The depth camera 104 is a camera capable of generating a depth information image including depth information, which is the distance to an imaging target. The depth information image includes depth information for each pixel. The depth camera 104 is sometimes called a depth camera. Known techniques can be used to measure the distance, such as triangulation and TOF (Time Of Flight). The depth camera 104 may be a stereo camera. The depth camera 104 may be built into the image processing device 100. The depth camera 104 may be a camera externally attached to the image processing device 100.

[0017] In this embodiment, the depth information image generated by the depth camera 104 has a lower resolution than the captured image generated by the imaging camera 102.

[0018] The image processing device 100 acquires an RGB image generated by the imaging camera 102 by capturing an image of the user 40 from the imaging camera 102. The image processing device 100 acquires a depth information image generated by the depth camera 104 by capturing an image of the user 40 from the depth camera 104. The depth information image generated by capturing an image of the user 40 may be an image in which the depth (three-dimensional shape) of the user 40 is expressed by discrete point cloud data.

[0019] The image processing device 100 may generate a noise-removed depth information image by removing the background portion from the depth information image, perform calibration so that the portion of the user 40 in the RGB image matches the portion of the user 40 in the noise-removed depth information image, and generate a superimposed image by superimposing the RGB image and the noise-removed depth information image.

[0020] The image processing device 100 may generate an avatar of the face of the user 40 based on the superimposed image. The image processing device 100 may determine, based on the superimposed image, a portion in which an object exists in front of the face of the user 40 within a region corresponding to the face of the user 40. The image processing device 100 may perform control so that, in the region of the RGB image corresponding to the face of the user 40 where an object exists in front of the face of the user 40, the corresponding portion of the RGB image is displayed, and in the region where no object exists in front of the face of the user 40, the corresponding portion of the avatar is displayed.

[0021] The image processing device 100 may perform control so that an RGB image and an avatar are displayed on a display provided in the image processing device 100. The image processing device 100 may transmit display data for displaying the RGB image and the avatar to another device, and perform control so that the RGB image and the avatar are displayed on a display of the other device.

[0022] Fig. 2 schematically illustrates another example of the avatar display system 10. In the example illustrated in Fig. 2, the avatar display system 10 includes an image processing device 100 and a communication device 200. The communication device 200 includes an imaging camera 202 and a depth camera 204. The communication device 200 may be, but is not limited to, a smartphone, a tablet terminal, a PC, or the like.

[0023] The imaging camera 202 may be similar to the imaging camera 102. The imaging camera 202 provides captured images to the communication device 200. The imaging camera 202 may be built into the communication device 200. The imaging camera 202 may also be a camera that is externally attached to the communication device 200.

[0024] The depth camera 204 may be the same as the depth camera 104. The imaging camera 202 is a camera capable of generating a depth information image including depth information, which is the distance to an imaging target. The depth camera 204 may be built into the communication device 200. The depth camera 204 may also be a camera externally attached to the communication device 200.

[0025] In this embodiment, the depth information image generated by the depth camera 204 has a lower resolution than the captured image generated by the imaging camera 202.

[0026] The image processing device 100 receives an RGB image generated by an imaging camera 202 capturing an image of the user 40 from the communication device 200 via the network 20. The image processing device 100 receives a depth information image generated by a depth camera 204 capturing an image of the user 40 from the communication device 200 via the network 20.

[0027] The network 20 may include the Internet. The network 20 may include a LAN (Local Area Network). The network 20 may include a mobile communication network. The mobile communication network may conform to any of the following communication methods: LTE (Long Term Evolution), 5G (5th Generation), 3G (3rd Generation), and 6G (6th Generation) or later.

[0028] The image processing device 100 may be connected to the network 20 by wire. The image processing device 100 may be connected to the network 20 by wireless. The image processing device 100 may be connected to the network 20 via a wireless base station. The image processing device 100 may be connected to the network 20 via a Wi-Fi (registered trademark) access point.

[0029] The communication device 200 may be connected to the network 20 by wire. The communication device 200 may be connected to the network 20 by wireless. The communication device 200 may be connected to the network 20 via a wireless base station. The communication device 200 may be connected to the network 20 via a Wi-Fi access point.

[0030] The image processing device 100 may generate a noise-removed depth information image by removing the background portion from the depth information image, perform calibration so that the portion of the user 40 in the RGB image matches the portion of the user 40 in the noise-removed depth information image, and generate a superimposed image by superimposing the RGB image and the noise-removed depth information image.

[0031] The image processing device 100 may generate an avatar of the face of the user 40 based on the superimposed image. The image processing device 100 may determine, based on the superimposed image, a portion in which an object exists in front of the face of the user 40 within a region corresponding to the face of the user 40. The image processing device 100 may perform control so that, in the region of the RGB image corresponding to the face of the user 40 where an object exists in front of the face of the user 40, the corresponding portion of the RGB image is displayed, and in the region where no object exists in front of the face of the user 40, the corresponding portion of the avatar is displayed.

[0032] The image processing device 100 may perform control so that an RGB image and an avatar are displayed on a display provided in the image processing device 100. The image processing device 100 may transmit display data for displaying the RGB image and the avatar to the communication device 200, and perform control so that the RGB image and the avatar are displayed on a display of the communication device 200. The image processing device 100 may transmit display data for displaying the RGB image and the avatar to a device other than the communication device 200, and perform control so that the RGB image and the avatar are displayed on a display of the device.

[0033] 3 is an explanatory diagram for roughly explaining avatar generation by image processing device 100. Here, a case where there is no object in front of the face of user 40 will be described.

[0034] The image processing device 100 generates a noise-removed depth information image 312 by removing background portions from the depth information image 310. For example, the image processing device 100 removes portions whose depth is deeper than a predetermined threshold from the depth information image 310. For example, the image processing device 100 removes background portions identified from the average depth of the depth information image 310.

[0035] The image processing device 100 calibrates the RGB image 320 so that the portion of the user 40 matches the portion of the user 40 in the noise-removed depth information image 312, and then superimposes the RGB image 320 and the noise-removed depth information image 312 to generate a superimposed image 330.

[0036] The image processing device 100 generates an avatar of the face of the user 40 based on the superimposed image 330. The image processing device 100 determines, based on the superimposed image 330, a portion in the area corresponding to the face of the user 40 where an object exists in front of the face of the user 40. In this example, it determines that no object exists.

[0037] The image processing device 100 may perform control to display a display image 340 in which the portion other than the face of the user 40 is a real-life image, and in which the portion of the face of the user 40 is an avatar 342 that moves in accordance with the movement of the face of the user 40. The image processing device 100 may perform control to display a display image 340 in which the portion of the RGB image 320 corresponding to the face of the user 40 is placed in a portion of the RGB image 320 other than the area corresponding to the face of the user 40, and the generated avatar 342 is placed in the portion of the RGB image 320 where the face of the user 40 is placed. By continuously performing such processing, it is possible to display a display image 340 including the avatar 342 that follows the movement of the user 40.

[0038] 4 is an explanatory diagram for roughly explaining avatar generation by image processing device 100. Here, a case where the hand of user 40 is in front of the face of user 40 will be described. The process up to generation of superimposed image 330 is the same as in FIG. 3.

[0039] The image processing device 100 generates an avatar of the face of the user 40 based on the superimposed image 330. The image processing device 100 determines, based on the superimposed image 330, a portion in the area corresponding to the face of the user 40 where an object exists in front of the face of the user 40. In this example, it is determined that an object exists.

[0040] The image processing device 100 may perform control to display a display image 340 in which the corresponding portion of the RGB image 320 is arranged in a portion of the RGB image 320 other than the region corresponding to the face of the user 40, the corresponding portion of the RGB image 320 is arranged in a portion of the region of the RGB image 320 corresponding to the face of the user 40 where an object exists in front of the face of the user 40, and the corresponding portion of the avatar 342 is arranged in a region where no object exists in front of the face of the user 40. In this way, when the hand of the user 40 is present in front of the face of the user 40, it is possible to display a display image 340 in which the corresponding portion of the RGB image 320 is arranged in a portion of the region corresponding to the face of the user 40 other than the region corresponding to the face of the user 40, the corresponding portion of the avatar 342 is arranged in a portion of the region corresponding to the face of the user 40 where the hand does not exist, and the corresponding portion of the RGB image 320 is arranged in a portion of the region corresponding to the face of the user 40 where the hand does not exist.

[0041] 5 shows an example of a portion 354 where a hand is present in a region 352 corresponding to the face of the user 40 in the display image 340. In FIG. 5, the region 352 is surrounded by a solid line, and the portion 354 is surrounded by a dashed line. The image processing device 100 may identify the region corresponding to the face of the user 40 by analyzing the superimposed image 330. The image processing device 100 may determine whether each pixel in the region corresponding to the face of the user 40 is a face or a hand by referring to the depth.

[0042] For example, the image processing device 100 calculates the average depth of the face of the user 40, and determines whether each pixel in a region corresponding to the face of the user 40 is a face or a hand based on the average depth. For example, the image processing device 100 determines that a pixel is a hand when the depth is shallower than the average depth by a predetermined threshold or more, and determines that a pixel is a face when the depth is not shallower. That is, the image processing device 100 determines that a part of the region corresponding to the face of the user 40 that is shallower than the average depth by a predetermined threshold or more is a part where a hand is present.

[0043] The image processing device 100 calculates the average depth of the face of the user 40 from the superimposed image 330 in a state where no object is present in front of the face of the user 40. The image processing device 100 analyzes the superimposed image 330 to identify an area corresponding to the face of the user 40, and calculates the average depth from the depth of each pixel in the identified area.

[0044] The image processing device 100 may calculate the average depth of the face of the user 40 from the superimposed image 330 in a state where an object is present in front of the face of the user 40. The image processing device 100 identifies an area corresponding to the face of the user 40 by analyzing the superimposed image 330, and calculates the average depth from the depth of each pixel in the identified area other than the part where the object is likely to exist. For example, the image processing device 100 divides the identified area into a group of pixels with a deep depth and a group of pixels with a shallow depth, and calculates the average depth from the depth of each pixel in the group of pixels with a deep depth.

[0045] The image processing device 100 may use the average depth of the entire face of the user 40 as the average depth of the face of the user 40. The image processing device 100 may use the average depth of a predetermined region of the face of the user 40 as the average depth of the face of the user 40. The predetermined region may be arbitrarily set to, for example, a region including the eyes, nose, and mouth of the user 40.

[0046] The image processing device 100 may determine whether each pixel in a region corresponding to the face of the user 40 is a face or a hand, for example, based on the depth of a point on the face of the user 40. As a specific example, the image processing device 100 determines whether each pixel in a region corresponding to the face of the user 40 is a face or a hand, based on the depth of the midpoint of the face of the user 40. For example, the image processing device 100 may determine the tip of the nose of the user 40 as the midpoint of the face of the user 40. The image processing device 100 may determine the midpoint of the region corresponding to the face of the user 40 as the midpoint of the face of the user 40. For example, the image processing device 100 determines that a pixel is a hand when the depth of the pixel is shallower than the depth of the midpoint by a predetermined threshold or more, and determines that a pixel is a face when the depth is not shallower. That is, the image processing device 100 determines that a part of the region corresponding to the face of the user 40 that is shallower than the depth of the midpoint by a predetermined threshold or more is a part where a hand is present. This reduces the processing load. However, if the depth of the midpoint of the face of the user 40 is not accurate, an error in determining whether or not an object exists in front of the face of the user 40, a so-called occlusion error, may occur. In contrast, as described above, by using the average depth of the face of the user 40, the occlusion error can be reduced.

[0047] 6 shows an example of the functional configuration of the image processing device 100. The image processing device 100 includes an acquisition unit 112, a superimposed image generation unit 114, a noise-removed image generation unit 116, a determination unit 118, an avatar generation unit 120, and a display control unit 122. Note that it is not essential for the image processing device 100 to include all of these units.

[0048] The acquisition unit 112 acquires an RGB image 320 of the user 40 that is generated by capturing an image of the user 40. The acquisition unit 112 may acquire the RGB image 320 generated by the imaging camera 102 from the imaging camera 102. The acquisition unit 112 may receive the RGB image 320 generated by the imaging camera 202 from the communication device 200.

[0049] The acquisition unit 112 acquires a depth information image 310 of the user 40 that is generated by capturing an image of the user 40. The acquisition unit 112 acquires the depth information image 310 that has a lower resolution than the RGB image 320. The acquisition unit 112 may acquire the depth information image 310 generated by the depth camera 104 from the depth camera 104. The acquisition unit 112 may receive the depth information image 310 generated by the depth camera 204 from the communication device 200.

[0050] The noise-removed image generation unit 116 generates a noise-removed depth information image 312 by removing background portions from the depth information image 310. For example, the noise-removed image generation unit 116 removes portions whose depth is deeper than a predetermined threshold from the depth information image 310. For example, the noise-removed image generation unit 116 removes background portions identified from the average depth of the depth information image 310.

[0051] The superimposed image generation unit 114 performs calibration to match the user's portion of the RGB image 320 with the user's portion of the noise-removed depth information image 312, and generates the superimposed image 330 by superimposing the RGB image 320 and the noise-removed depth information image 312.

[0052] The superimposed image generation unit 114 may perform calibration so that the user's portion of the RGB image 320 matches the user's portion of the noise-reduced depth information image 312, depending on the difference in resolution between the RGB image 320 and the depth information image 310. For example, the superimposed image generation unit 114 enlarges the noise-reduced depth information image 312 by a magnification factor depending on the difference in resolution between the RGB image 320 and the depth information image 310. As another example, the superimposed image generation unit 114 may reduce the RGB image 320 by a magnification factor depending on the difference in resolution between the RGB image 320 and the depth information image 310.

[0053] The superimposed image generation unit 114 may adjust at least one of the size of the noise-removed depth information image 312 and the positional relationship between the RGB image 320 and the noise-removed depth information image 312 so as to increase the degree of correspondence between the user's portion of the RGB image 320 and the user's portion of the noise-removed depth information image 312, and may generate the superimposed image 330 by superimposing the RGB image 320 and the noise-removed depth information image 312.

[0054] For example, the superimposed image generation unit 114 adjusts the size of the noise-removed depth information image 312 so as to increase the degree of match between the user's portion in the RGB image 320 and the user's portion in the noise-removed depth information image 312. As a specific example, the superimposed image generation unit 114 calculates the degree of match between the user's portion in the RGB image 320 and the user's portion in the noise-removed depth information image 312 while gradually increasing the size of the noise-removed depth information image 312, and superimposes the RGB image 320 and the noise-removed depth information image 312 with the size that results in the highest degree of match.

[0055] For example, the superimposed image generation unit 114 adjusts the positional relationship between the RGB image 320 and the noise-removed depth information image 312 so that the degree of match between the user's portion in the RGB image 320 and the noise-removed depth information image 312 becomes higher. The superimposed image generation unit 114 adjusts, for example, the relative position between the RGB image 320 and the noise-removed depth information image 312. The superimposed image generation unit 114 adjusts, for example, the relative angle between the RGB image 320 and the noise-removed depth information image 312. The superimposed image generation unit 114 adjusts, for example, the relative position and relative angle between the RGB image 320 and the noise-removed depth information image 312. As a specific example, the superimposed image generation unit 114 enlarges the noise-removed depth information image 312 by a magnification corresponding to the difference in resolution between the RGB image 320 and the depth information image 310, and then calculates the degree of correspondence between the user's part in the RGB image 320 and the user's part in the noise-removed depth information image 312 while gradually changing the positional relationship between the RGB image 320 and the noise-removed depth information image 312, and superimposes the RGB image 320 and the noise-removed depth information image 312 at the positional relationship that results in the highest degree of correspondence.

[0056] For example, the superimposed image generation unit 114 adjusts both the size of the noise-removed depth information image 312 and the positional relationship between the RGB image 320 and the noise-removed depth information image 312 so as to increase the degree of match between the user's portion in the RGB image 320 and the user's portion in the noise-removed depth information image 312. As a specific example, the superimposed image generation unit 114 calculates the degree of match between the user's portion in the RGB image 320 and the user's portion in the noise-removed depth information image 312 while gradually increasing the size of the noise-removed depth information image 312 or changing the positional relationship between the RGB image 320 and the noise-removed depth information image 312, and superimposes the RGB image 320 and the noise-removed depth information image 312 at the size and positional relationship between the RGB image 320 and the noise-removed depth information image 312 that results in the highest degree of match.

[0057] Depending on the positional relationship between the imaging camera 102 and the depth camera 104, if the noise-removed depth information image 312 is simply enlarged and superimposed on the RGB image 320, the portion of the user in the RGB image 320 may not accurately match the portion of the user in the noise-removed depth information image 312. In such cases, it may be impossible to accurately generate the avatar 342 or accurately recognize an object in front of the face of the user 40; however, by having the superimposed image generation unit 114 perform the above-described calibration, it is possible to reduce such possibilities.

[0058] The superimposed image generation unit 114 may perform calibration so that the user's portion of the RGB image 320 matches the user's portion of the depth information image 310 acquired by the acquisition unit 112, rather than the noise-removed depth information image 312, and may generate the superimposed image 330 by superimposing the RGB image 320 and the depth information image 310.

[0059] The determination unit 118 determines, based on the superimposed image 330, a portion where an object exists in front of the face of the user 40 within the region corresponding to the face of the user 40. The determination unit 118 may calculate an average depth of the user's face based on the superimposed image 330, and determine, based on the average depth, a portion where an object exists in front of the user's face.

[0060] For example, the determination unit 118 calculates the average depth of the face of the user 40, and determines whether each pixel in the area corresponding to the face of the user 40 is a face or an object present in front of the face based on the average depth. For example, the determination unit 118 determines that a pixel is an object present in front of the face when the depth of the pixel is shallower than the average depth by a predetermined threshold or more, and determines that the pixel is a face when the depth is not shallower. That is, the determination unit 118 determines that a portion of the area corresponding to the face of the user 40 whose depth is shallower than the average depth by a predetermined threshold or more is a portion where an object exists.

[0061] The determination unit 118 calculates the average depth of the face of the user 40 from the superimposed image 330 in a state where no object exists in front of the face of the user 40. The determination unit 118 analyzes the superimposed image 330 to identify an area corresponding to the face of the user 40, and calculates the average depth from the depth of each pixel in the identified area.

[0062] The determination unit 118 may calculate the average depth of the face of the user 40 from the superimposed image 300 in a state where an object is present in front of the face of the user 40. The determination unit 118 analyzes the superimposed image 330 to identify an area corresponding to the face of the user 40, and calculates the average depth from the depth of each pixel in the identified area other than the part where the object is likely to exist. The image processing device 100, for example, divides the identified area into a group of pixels with a deep depth and a group of pixels with a shallow depth, and calculates the average depth from the depth of each pixel in the group of pixels with a deep depth.

[0063] The determination unit 118 may use the average depth of the entire face of the user 40 as the average depth of the face of the user 40. The determination unit 118 may use the average depth of a predetermined region of the face of the user 40 as the average depth of the face of the user 40. The predetermined region may be arbitrarily set to, for example, a region including the eyes, nose, and mouth of the user 40.

[0064] Note that the determination unit 118 may determine whether each pixel in a region corresponding to the face of the user 40 is a face or a hand, for example, based on the depth of a point on the face of the user 40. As a specific example, the determination unit 118 determines whether each pixel in a region corresponding to the face of the user 40 is a face or a hand, based on the depth of the midpoint of the face of the user 40. For example, the determination unit 118 determines the tip of the nose of the user 40 as the midpoint of the face of the user 40. The determination unit 118 may also determine the midpoint of the region corresponding to the face of the user 40 as the midpoint of the face of the user 40.

[0065] The avatar generation unit 120 generates an avatar 342 of the face of the user 40 based on the superimposed image 330. For example, the avatar generation unit 120 generates a 3D model of the face of the user 40 using the noise-removed depth information image 312 from the superimposed image 330, generates texture data of the face of the user 40 using the RGB image 320 from the superimposed image 330, and performs a process of pasting the texture data onto the 3D model, thereby generating the avatar 342. The avatar generation unit 120 may generate the avatar 342 from the superimposed image 330 using a method other than the above.

[0066] The display control unit 122 performs control to display a display image 340 in accordance with the determination result by the determination unit 118. If the determination unit 118 determines that no object exists in front of the face of the user 40, the display control unit 122 performs control to display a display image 340 in which the corresponding portion of the RGB image 320 is placed in a portion of the RGB image 320 other than the area corresponding to the face of the user 40, and an avatar 342 generated by the avatar generation unit 120 is placed in the portion of the RGB image 320 where the face of the user 40 is located.

[0067] When the determination unit 118 determines that an object exists in front of the face of the user 40, the display control unit 122 may perform control to display a display image 340 in which the corresponding portion of the RGB image 320 is arranged in a portion of the RGB image 320 other than the region corresponding to the face of the user 40, the corresponding portion of the RGB image 320 is arranged in a portion of the region of the RGB image 320 corresponding to the face of the user 40 where an object exists in front of the face of the user 40, and the corresponding portion of the avatar 342 is arranged in a region where no object exists in front of the face of the user 40. For example, when the hand of the user 40 is in front of the face of the user 40, the display control unit 122 performs control to display a display image 340 in which the portion of the hand of the user 40 in the RGB image 320 is arranged in the portion of the region corresponding to the face of the user 40 where the hand of the user 40 exists in front of the face of the user 40.

[0068] When it is determined that user 40 cannot be detected within the imaging range while controlling the display of avatar 342 by placing it in the area of ​​the user's face in RGB image 320, display control unit 122 may control the display of the displayed avatar 342 to be fixed. If user 40 within the imaging range is overlooked, the avatar 342 may disappear and a live-action image of the face of user 40 may be displayed, which may be inconvenient. However, by controlling the display control unit 122 to display the avatar 342 in a fixed position as described above, the avatar 342 can be fixed in its final position, preventing the avatar 342 from disappearing and the user's face from being displayed.

[0069] 7 schematically illustrates an example of the hardware configuration of a computer 1200 functioning as the image processing device 100 or the communication device 200. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "units" of the device according to the present embodiment, or can cause the computer 1200 to perform operations associated with the device according to the present embodiment or one or more "units," and / or can cause the computer 1200 to perform a process according to the present embodiment or steps of the process. Such a program can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0070] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communications interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and legacy input / output units such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0071] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into the graphics controller itself, and causes the image data to be displayed on the display device 1218.

[0072] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0073] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0074] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and causes cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing operations or processing of information in accordance with the use of the computer 1200.

[0075] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded into the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0076] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and may perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0077] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0078] The above-described programs or software modules may be stored in a computer-readable storage medium on or near the computer 1200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0079] The blocks in the flowcharts and block diagrams in the present embodiments may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, including integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0080] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray disc, memory stick, integrated circuit card, etc.

[0081] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0082] Computer-readable instructions may be provided locally or over a wide area network (WAN) such as a local area network (LAN), the Internet, etc. to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or programmable circuitry, such that the processor or programmable circuitry executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0083] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0084] It should be noted that the execution order of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order. [Explanation of symbols]

[0085] 10 avatar display system, 20 network, 40 user, 100 image processing device, 102 imaging camera, 104 depth camera, 112 acquisition unit, 114 superimposed image generation unit, 116 noise-removed image generation unit, 118 determination unit, 120 avatar generation unit, 122 display control unit, 200 communication device, 202 imaging camera, 204 depth camera, 310 depth information image, 312 noise-removed depth information image, 320 RGB image, 330 superimposed image, 340 display image, 342 avatar, 352 region, 354 part, 1200 computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphics controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1230 ROM, 1240 I / O chip

Claims

1. an acquisition unit that acquires an RGB image and a depth information image of the user having a lower resolution than the RGB image, both of which are generated by capturing an image of the user; a noise-removed image generating unit that generates a noise-removed depth information image by removing a background portion from the depth information image; a superimposed image generation unit that performs calibration so that the user's portion of the RGB image matches the user's portion of the noise-removed depth information image, and generates a superimposed image by superimposing the RGB image and the noise-removed depth information image; an avatar generation unit that generates an avatar of the user's face based on the superimposed image; An image processing device comprising:

2. The image processing device according to claim 1, wherein the superimposed image generation unit performs calibration so as to match the user's portion of the RGB image with the user's portion of the noise-removed depth information image in accordance with a difference in resolution between the RGB image and the depth information image.

3. 2. The image processing device according to claim 1, wherein the superimposed image generation unit adjusts at least one of the size of the noise-removed depth information image and the positional relationship between the RGB image and the noise-removed depth information image so as to increase the degree of correspondence between the user's portion of the RGB image and the user's portion of the noise-removed depth information image, and generates the superimposed image by superimposing the RGB image and the noise-removed depth information image.

4. a determination unit that determines, based on the superimposed image, a portion in which an object exists in front of the user's face within a region corresponding to the user's face; a display control unit that controls to display the avatar by arranging it in a portion of the user's face in the RGB image, the display control unit controlling to display a display image in which the corresponding portion of the RGB image is arranged in a portion of an area corresponding to the user's face where an object exists in front of the user's face, and the corresponding portion of the avatar is arranged in an area where no object exists in front of the user's face; The image processing device according to claim 1 , comprising:

5. The image processing device according to claim 4, wherein the display control unit controls the display image to display the user's hand portion of the RGB image in a portion of the area corresponding to the user's face where the user's hand is located in front of the user's face.

6. The image processing device according to claim 5 , wherein the determination unit calculates an average depth of the user's face based on the superimposed image, and determines a portion where an object exists in front of the user's face based on the average depth.

7. The image processing device according to claim 6, wherein the determination unit determines that a portion of the region corresponding to the user's face that is shallower than the average depth by a predetermined threshold or more is a portion where an object is present in front of the user's face.

8. 5. The image processing device of claim 4, wherein the display control unit controls the avatar to be displayed fixedly when it is determined that the user cannot be detected within the imaging range while controlling the avatar to be positioned and displayed in the area of ​​the user's face in the RGB image.

9. A program for causing a computer to function as the image processing device according to any one of claims 1 to 3.

10. 1. A computer-implemented image processing method comprising: an acquisition step of acquiring an RGB image and a depth information image of the user having a lower resolution than the RGB image, the RGB image and the depth information image being generated by capturing an image of the user; generating a noise-removed depth information image by removing a background portion from the depth information image; a superimposed image generating step of performing calibration so that the user's portion of the RGB image matches the user's portion of the noise-removed depth information image, and generating a superimposed image by superimposing the RGB image and the noise-removed depth information image; an avatar generation step of generating an avatar of the user's face based on the superimposed image; An image processing method comprising:

Images