IMAGE PROCESSING APPARATUS, PROGRAM, AND IMAGE PROCESSING METHOD

Abstract

An image processing device, a program, and an image processing method are provided that display an avatar that follows the movements of a user. [Solution] The image processing device 100 includes a first calculation unit that analyzes imaging data capturing an image of a user's face and calculates feature amounts including the movement direction and movement speed of each of a plurality of feature points on the user's face; a second calculation unit that calculates weighted feature amounts by applying weighting values ​​corresponding to parts of the user's face to each of the feature amounts of at least a portion of the plurality of feature points on the user's face; an avatar generation unit that generates an avatar that moves in accordance with the movement of the user's face based on the weighted feature amounts of each of the plurality of feature points; and a display control unit that controls the avatar generated by the avatar generation unit to be displayed superimposed on the part of the user's face in the imaging data.

Description

[Technical field]

[0001] The present invention relates to an image processing device, a program, and an image processing method. [Background technology]

[0002] Patent Document 1 describes a technique for displaying an avatar that follows the movements of a user. [Prior art document] [Patent documents] [Patent Document 1] JP 2023-094549 A Summary of the Invention [Means for solving the problem]

[0003] According to an embodiment of the present invention, there is provided an image processing device. The image processing device may include a first calculation unit that analyzes imaging data of a user's face and calculates a feature amount including a moving direction and a moving speed of each of a plurality of feature points of the user's face. The image processing device may include a second calculation unit that calculates a weighted feature amount by applying a weighting value corresponding to a part of the face to each of the feature amounts of at least a part of the plurality of feature points of the user's face. The image processing device may include an avatar generation unit that generates an avatar that moves in accordance with the movement of the user's face, and generates the avatar based on the weighted feature amount of each of the plurality of feature points. The image processing device may include a display control unit that controls to display the avatar generated by the avatar generation unit superimposed on the part of the user's face in the imaging data.

[0004] In the image processing device, the second calculation unit may calculate the weighted feature amount by applying a weighting value corresponding to a part of the face to the moving speed included in the feature amount of each of the at least some of the plurality of feature points, and the avatar generation unit may generate the avatar in which each part changes by moving each of the at least some of the plurality of feature points according to the moving speed to which the weighting value has been applied. The image processing device may include a range storage unit that stores a movable range of each of the plurality of feature points of the user's face, and the avatar generation unit may generate the avatar in which each part changes by moving each of the at least some of the plurality of feature points within the movable range according to the moving speed to which the weighting value has been applied.

[0005] Any of the image processing devices may include a mode management unit that switches between a normal mode and an eye emphasis mode that emphasizes the eyes of the user, and when the normal mode is set, the avatar generation unit may generate the avatar based on the imaging data, and when the eye emphasis mode is set, the second calculation unit may calculate the weighted feature amount by applying a weighting value corresponding to the eyes to each of the feature amounts of a plurality of feature points corresponding to the user's eyes among the plurality of feature points of the user's face, and the avatar generation unit may generate the avatar using the weighted feature amounts of the plurality of feature points corresponding to the user's eyes. The mode management unit may switch between the normal mode, the eye emphasis mode, and an expression emphasis mode that emphasizes the user's expression, and when the expression emphasis mode is set, the second calculation unit may calculate weighted feature amounts by applying weighting values ​​corresponding to parts of the face to the feature amounts of each of a plurality of feature points among the plurality of feature points of the user's face that correspond to the user's eyes, eyebrows, cheeks, and mouth, and the avatar generation unit may generate the avatar using the weighted feature amounts of the plurality of feature points that correspond to the user's eyes, eyebrows, cheeks, and mouth.

[0006] Any of the image processing devices may include a mode management unit that switches between a normal mode and an expression enhancement mode that enhances the user's expression, and when the normal mode is set, the avatar generation unit may generate the avatar based on the imaging data, and when the expression enhancement mode is set, the second calculation unit may calculate weighted features by applying weighting values ​​corresponding to parts of the face to the feature amounts of each of a plurality of feature points corresponding to the user's eyes, eyebrows, cheeks, and mouth among the plurality of feature points of the user's face, and the avatar generation unit may generate the avatar based on the weighted features of the plurality of feature points corresponding to the user's eyes, eyebrows, cheeks, and mouth and the feature amounts of a plurality of feature points other than the user's eyes, eyebrows, cheeks, and mouth.

[0007] According to one embodiment of the present invention, there is provided a program for causing a computer to function as the image processing device.

[0008] According to one embodiment of the present invention, there is provided an image processing method executed by a computer. The image processing method may include a first calculation step of analyzing imaging data capturing an image of a user's face and calculating a feature amount including a moving direction and a moving speed of each of a plurality of feature points of the user's face. The image processing method may include a second calculation step of calculating a weighted feature amount by applying a weighting value corresponding to a part of the face to each of the feature amounts of at least a part of the plurality of feature points of the user's face. The image processing method may include an avatar generation step of an avatar generation unit that generates an avatar that moves in accordance with the movement of the user's face, and generating the avatar based on the weighted feature amount of each of the plurality of feature points. The image processing method may include a display control step of controlling to display the avatar generated by the avatar generation unit so as to be superimposed on the part of the user's face in the imaging data.

[0009] The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0010] [Figure 1] 1 illustrates a schematic diagram of an example avatar display system 10. [Diagram 2] 2 illustrates a schematic diagram of another example of an avatar display system 10. [Diagram 3] 2 illustrates an example of a functional configuration of the image processing device 100. [Figure 4] FIG. 11 is an explanatory diagram for explaining eye emphasis. [Diagram 5] FIG. 11 is an explanatory diagram for explaining facial expression emphasis. [Figure 6] 1 shows an example of a processing flow by the image processing device 100. [Figure 7] An example of the hardware configuration of a computer 1200 functioning as the image processing device 100 or the communication device 200 is shown in schematic form. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0012] By making the avatar perform a different movement from the user's facial movement, rather than the user's facial movement itself, the convenience of the user may be improved. For example, some users may wish to prevent others from understanding their facial expressions, and such a wish can be met. For example, for a user who has a facial defect and has difficulty expressing himself / herself, making the avatar perform a movement that eliminates the defect can enable the user to express the same facial expression as a person without a facial defect. Also, making the avatar perform a movement that emphasizes the user's facial movement can give the other person a good impression or an interesting experience.

[0013] Fig. 1 illustrates an example of an avatar display system 10. In the example illustrated in Fig. 1, the avatar display system 10 includes an image processing device 100. The image processing device 100 includes an imaging unit 102. The image processing device 100 may be, but is not limited to, a smartphone, a tablet terminal, a PC (Personal Computer), or the like.

[0014] The imaging unit 102 provides imaging data generated by imaging an imaging target to the image processing device 100. The imaging unit 102 has an imaging camera. The imaging data may include an imaging image generated by the imaging camera. The imaging image may be an RGB image. The imaging image may be a moving image. The imaging image may be a still image captured continuously.

[0015] The imaging unit 102 may further include a depth camera. The depth camera is a camera capable of generating a depth information image including depth information, which is the distance to an imaging target. The depth information image includes depth information for each pixel. The depth camera may also be called a depth camera. A known technique may be used to measure the distance, and for example, LiDAR (Light Detection And Ranging), a triangulation method, or a TOF (Time Of Flight) method may be used. The imaging data may include the depth information image.

[0016] The imaging unit 102 may include a stereo camera. The stereo camera may generate a captured image and a depth information image.

[0017] The imaging unit 102 may be built into the image processing device 100. The imaging unit 102 may be attached externally to the image processing device 100.

[0018] The image processing device 100 may acquire imaging data of the face of the user 40 captured by the imaging unit 102 from the imaging unit 102, and generate an avatar of the face of the user 40 using the imaging data. For example, the image processing device 100 generates an avatar of the face of the user 40 using the captured image and depth information image included in the imaging data. The depth information image generated by capturing an image of the face of the user 40 may be an image in which the depth (three-dimensional shape) of the user 40 is expressed by discrete point cloud data.

[0019] The image processing device 100 generates an avatar by, for example, generating a 3D model of the face of the user 40 using a depth information image, generating texture data of the face of the user 40 using a captured image, and performing a process of pasting the texture data onto the 3D model. The image processing device 100 may generate an avatar that moves in accordance with the movement of the face of the user 40 by continuously generating avatars from continuously acquired captured image data.

[0020] The image processing device 100 according to the present embodiment may generate an avatar in which a part or the whole of the face of the user 40 is emphasized. For example, the image processing device 100 generates an avatar in which the facial expression of the user 40 is emphasized. For example, the image processing device 100 generates an avatar in which the eyes of the user 40 are emphasized.

[0021] The image processing device 100 may perform control so as to display a display image in which an avatar is arranged in a region corresponding to the face of the user 40 in an RGB image included in the imaging data. For example, the image processing device 100 may perform control so as to display the display image on a display provided in the image processing device 100. For example, the image processing device 100 may perform control so as to transmit the display image to another device and display the display image on the display of the other device.

[0022] The image processing device 100 may analyze imaging data capturing an image of the face of the user 40 and calculate a feature amount for each of a plurality of feature points on the face of the user 40. The feature amount may include a moving direction and a moving speed of the feature point. The image processing device 100 may calculate a weighted feature amount by applying a weighting value corresponding to a part of the face to each of the feature amounts of at least a part of the plurality of feature points on the face of the user 40. The image processing device 100 may generate an avatar by using the weighted feature amount for feature points to which a weighting value has been applied among the plurality of feature points on the face of the user 40, and using the feature amount for feature points to which a weighting value has not been applied.

[0023] For example, the image processing device 100 calculates weighted features by applying weighting values ​​corresponding to parts of the face to the movement speeds included in the features of at least a portion of a plurality of feature points on the face of the user 40, and generates an avatar in which each part changes by moving each of the at least a portion of the feature points according to the movement speed to which the weighting value has been applied.

[0024] For example, the image processing device 100 generates an avatar in which the movement of a part of the face of the user 40 is emphasized. As a specific example, the image processing device 100 calculates a weighted feature amount by applying a weight value corresponding to the eyes to the movement speed included in each feature amount of a plurality of feature points corresponding to the eyes of the user 40 on the face of the user 40, and generates an avatar in which the movement of the eyes is emphasized by moving each of the plurality of feature points corresponding to the eyes according to the movement speed to which the weight value has been applied.

[0025] For example, the image processing device 100 generates an avatar in which the movements of multiple parts of the face of the user 40 are emphasized. As a specific example, a weighted feature amount is calculated by applying a weight value corresponding to each part to the movement speed included in each feature amount of multiple feature points corresponding to the eyes, nose, mouth, and cheeks of the face of the user 40, and an avatar in which the facial expression is emphasized is generated by moving each of the multiple feature points of the eyes, nose, mouth, and cheeks of the face of the user 40 according to the movement speed to which the weight value has been applied. Among the parts of the face of the user 40, the parts whose movement of the avatar is emphasized may be arbitrarily set.

[0026] In this way, by emphasizing a part or the whole of the face of the user 40, for example, it is possible to prevent the facial expression of the user from being revealed to the other party with whom the user is communicating via the avatar. It is also possible to give the other party a good impression and provide an interesting experience. It is also possible for a user 40 who has some kind of facial disability and has difficulty expressing himself / herself to express his / her facial expressions in the same way as a person without a disability.

[0027] Fig. 2 illustrates a schematic diagram of another example of the avatar display system 10. In the example illustrated in Fig. 2, the avatar display system 10 includes an image processing device 100 and a communication device 200. The communication device 200 includes an imaging unit 202. The communication device 200 may be, but is not limited to, a smartphone, a tablet terminal, a PC (Personal Computer), or the like.

[0028] The imaging unit 202 may be similar to the imaging unit 102. The imaging unit 202 provides the communication device 200 with imaging data generated by imaging an imaging target. The imaging unit 202 has an imaging camera. The imaging data may include an imaging image generated by the imaging camera. The imaging image may be an RGB image. The imaging image may be a moving image. The imaging image may be a still image captured continuously. The imaging unit 202 may further have a depth camera. The imaging unit 202 may have a stereo camera. The stereo camera may generate the imaging image and the depth information image. The imaging unit 202 may be built in the communication device 200. The imaging unit 202 may be externally attached to the communication device 200.

[0029] The image processing device 100 receives imaging data generated by the imaging unit 202 capturing an image of the user 40 from the communication device 200 via the network 20 .

[0030] The network 20 may include the Internet. The network 20 may include a LAN (Local Area Network). The network 20 may include a mobile communication network. The mobile communication network may conform to any of the following communication methods: the LTE (Long Term Evolution) communication method, the 5G (5th Generation) communication method, the 3G (3rd Generation) communication method, and the 6G (6th Generation) communication method or later.

[0031] The image processing device 100 may be connected to the network 20 by wire. The image processing device 100 may be connected to the network 20 by wireless. The image processing device 100 may be connected to the network 20 via a wireless base station. The image processing device 100 may be connected to the network 20 via a Wi-Fi (registered trademark) access point.

[0032] The communication device 200 may be connected to the network 20 by wire. The communication device 200 may be connected to the network 20 by wireless. The communication device 200 may be connected to the network 20 via a wireless base station. The communication device 200 may be connected to the network 20 via a Wi-Fi access point.

[0033] The image processing device 100 may generate an avatar of the face of the user 40 by using the imaging data received from the communication device 200. The image processing device 100 may control to display a display image in which an avatar is arranged in a region corresponding to the face of the user 40 in an RGB image included in the imaging data. For example, the image processing device 100 may control to display the display image on a display provided in the image processing device 100. The image processing device 100 may control to transmit the display image to the communication device 200 and display the display image on the display of the communication device 200. The image processing device 100 may control to transmit the display image to a device other than the communication device 200 and display the display image on the display of the device.

[0034] 3 illustrates an example of a functional configuration of the image processing device 100. The image processing device 100 includes an imaging data acquisition unit 112, a first calculation unit 114, a second calculation unit 116, an avatar generation unit 118, a mode management unit 120, a range storage unit 122, and a display control unit 124. It is not essential that the image processing device 100 includes all of these units.

[0035] The imaging data acquisition unit 112 acquires imaging data capturing an image of the face of the user 40. For example, the imaging data acquisition unit 112 acquires imaging data generated by the imaging unit 102 from the imaging unit 102. For example, the imaging data acquisition unit 112 receives imaging data generated by the imaging unit 202 from the communication device 200.

[0036] The first calculation unit 114 analyzes the imaging data acquired by the imaging data acquisition unit 112 and calculates a feature amount of each of a plurality of feature points on the face of the user 40. The feature amount includes a moving direction and a moving speed of the feature point. The first calculation unit 114 may calculate the feature amount of each of the plurality of feature points by using a depth information image included in the imaging data.

[0037] The second calculation unit 116 calculates weighted feature amounts by applying weight values ​​corresponding to face parts to the feature amounts of at least some of the feature points calculated by the first calculation unit 114. The weight values ​​corresponding to face parts may be different values ​​depending on the face parts. For example, the weight value corresponding to the eyes and the weight value corresponding to the mouth may be different values. The weight values ​​corresponding to the face parts may be the same value. For example, the weight value corresponding to the eyes and the weight value corresponding to the mouth may be the same value. The weight values ​​corresponding to the face parts may be arbitrarily set or may be changeable.

[0038] The weighting value corresponding to the facial part may be appropriately changeable. The weighting value may be changeable, for example, by a slide bar displayed on the display of the image processing device 100. The weighting value may be configured to be changeable by a method other than a slide bar. The second calculation unit 116 may calculate a weighted feature amount by applying a weighting value corresponding to the facial part and set at that time to each feature amount of at least a part of the plurality of feature points calculated by the first calculation unit 114. The range of the weighting value may be different or the same for each facial part. An example of the range of the weighting value is 1.0 to 1.5, but is not limited thereto.

[0039] The second calculation unit 116 may calculate a weighted feature amount by applying a weighting value corresponding to a part of the face to the movement speed included in each feature amount of at least a portion of the multiple feature points calculated by the first calculation unit 114.

[0040] The avatar generation unit 118 generates an avatar that moves in accordance with the movement of the face of the user 40. For example, the avatar generation unit 118 generates a 3D model of the face of the user 40 using a depth information image included in the imaging data, generates texture data of the face of the user 40 using a captured image included in the imaging data, and performs a process of pasting the texture data onto the 3D model, thereby generating an avatar. By continuously performing such a process, it is possible to generate an avatar that moves in accordance with the movement of the face of the user 40.

[0041] The avatar generation unit 118 further generates an avatar that emphasizes the movement of a part or the whole of the face of the user 40. The avatar generation unit 118 generates an avatar based on the weighted feature amounts of each of the plurality of feature points calculated by the second calculation unit 116. The avatar generation unit 118 may generate an avatar in which each part changes by moving each of the plurality of feature points for which the second calculation unit 116 has calculated the weighted feature amount according to a moving speed to which a weighted value is applied.

[0042] For example, when generating an avatar that emphasizes the eye movement of the user 40, the second calculation unit 116 calculates a weighted feature amount by applying a weight value corresponding to the eyes to each feature amount of a plurality of feature points corresponding to the eyes of the user 40 among a plurality of feature points on the face of the user 40. Then, the avatar generation unit 118 generates an avatar using the weighted feature amount of the plurality of feature points corresponding to the eyes of the user 40. The avatar generation unit 118 generates an avatar in which the parts other than the eyes of the user 40 move in accordance with the movement of the face of the user 40, and the parts of the eyes of the user 40 move according to a movement speed increased by the weight value. By increasing the movement speed, the amount of movement also increases as a result, and the movement of the eyes is emphasized.

[0043] The avatar generation unit 118 may generate an avatar using BlendShape. For example, the avatar generation unit 118 generates an avatar whose facial expression changes by adjusting a shape key based on an avatar generated by attaching texture data to a 3D model of the face of the user 40. The avatar generation unit 118 may generate an avatar that moves in accordance with the movement of the face of the user 40 by adjusting the shape key according to the feature amount calculated by the first calculation unit 114. The avatar generation unit 118 may also generate an avatar that emphasizes the movement of the user 40 by adjusting the shape key according to the weighted feature amount calculated by the second calculation unit 116.

[0044] It is not necessary for the avatar generation unit 118 to generate a face model using all of the depth data of the face of the user 40. The avatar generation unit 118 may generate an avatar using one or more depth data selected from the depth data corresponding to the eyes, the depth data corresponding to the nose, and the depth data corresponding to the mouth, among the depth data of the face. As the depth data corresponding to the eyes of the user 40, depth data of one or more feature points around the eyes can be used, and the number of the feature points is not particularly limited. In addition, the one or more feature points around the eyes of the user 40 may include feature points corresponding to the eyebrows. As the depth data corresponding to the nose of the user 40, depth data of one or more feature points around the nose can be used, and the number of the feature points is not particularly limited. As the depth data corresponding to the mouth of the user 40, depth data of one or more feature points around the mouth can be used, and the number of the feature points is not particularly limited. In addition, the one or more feature points around the mouth of the user 40 may include a feature point corresponding to the chin. By generating an avatar using data on at least one of the eyes, nose, and mouth, which are easy to recognize in terms of position and orientation and are easy to express facial features, it is possible to reduce the amount of avatar data compared to an avatar generated using data on the entire face while maintaining a person's features, thereby speeding up image processing.

[0045] The method of generating a 3D model of the face of user 40 from one or more of the depth data corresponding to the eyes, the depth data corresponding to the nose, and the depth data corresponding to the mouth is not particularly limited, but the avatar generation unit 118 may generate a 3D model of the face of user 40 by using a learning model that is machine-learned using a set of various 3D face models and one or more of the depth data corresponding to the eyes, the depth data corresponding to the nose, and the depth data corresponding to the mouth of the 3D models as training data.

[0046] The mode management unit 120 manages modes related to avatar emphasis. Examples of the modes include a normal mode, an eye emphasis mode for emphasizing the eyes of the user 40, a mouth emphasis mode for emphasizing the mouth of the user 40, and an expression emphasis mode for emphasizing the facial expression of the user 40.

[0047] The mode management unit 120 switches the mode. For example, the mode management unit 120 switches the mode according to an instruction from the user 40. When the normal mode and the eye emphasis mode are prepared as the modes, the mode management unit 120 switches between the normal mode and the eye emphasis mode. When the normal mode and the facial expression emphasis mode are prepared as the modes, the mode management unit 120 switches between the normal mode and the facial expression emphasis mode. When the normal mode and the mouth emphasis mode are prepared as the modes, the mode management unit 120 switches between the normal mode and the mouth emphasis mode. When the normal mode and two or more of the eye emphasis mode, the mouth emphasis mode, and the facial expression emphasis mode are prepared, the mode management unit 120 switches between these modes.

[0048] When the normal mode is set, the avatar generation unit 118 may generate an avatar based on the imaging data acquired by the imaging data acquisition unit 112. That is, the avatar generation unit 118 may generate a 3D model of the face of the user 40 using the depth information image, generate texture data of the face of the user 40 using the captured image, and generate an avatar that moves in accordance with the movement of the face of the user 40 by continuously performing a process of pasting the texture data onto the 3D model. In the normal mode, no emphasis is performed.

[0049] When the eye emphasis mode is set, the second calculation unit 116 calculates weighted feature amounts by applying weighting values ​​corresponding to the eyes to the feature amounts of a plurality of feature points corresponding to the eyes of the user 40 among a plurality of feature points on the face of the user 40, and the avatar generation unit 118 generates an avatar using the weighted feature amounts of the plurality of feature points corresponding to the eyes of the user 40. The avatar generation unit 118 may generate an avatar by applying weighted feature amounts to a plurality of feature points corresponding to the eyes of an avatar generated based on the imaging data.

[0050] When the mouth emphasis mode is set, the second calculation unit 116 calculates weighted feature amounts by applying weighting values ​​corresponding to the mouth to the feature amounts of multiple feature points corresponding to the mouth of the user 40 among multiple feature points on the face of the user 40, and the avatar generation unit 118 generates an avatar using the weighted feature amounts of the multiple feature points corresponding to the mouth of the user 40. The avatar generation unit 118 may generate an avatar by applying weighted feature amounts to multiple feature points corresponding to the mouth of an avatar generated based on imaging data.

[0051] When the facial expression emphasis mode is set, the second calculation unit 116 may calculate weighted feature amounts by applying weighting values ​​corresponding to facial parts to feature amounts of a plurality of feature points corresponding to the user's eyes, eyebrows, cheeks, and mouth among a plurality of feature points on the face of the user 40, and the avatar generation unit 118 may generate an avatar using the weighted feature amounts of the plurality of feature points corresponding to the eyes, eyebrows, cheeks, and mouth of the user 40. The avatar generation unit 118 may generate an avatar by applying the weighted feature amounts to a plurality of feature points corresponding to the eyes, eyebrows, cheeks, and mouth of an avatar generated based on imaging data.

[0052] The range storage unit 122 stores the movable range of each of the multiple feature points on the face of the user 40. The movable range of a feature point may indicate the movable range of the feature point from a reference position on the face of the user 40. The movable range of each of the multiple feature points is set based on a range that is not unnatural for a human face. For example, if the feature point of the mouth moves to the position of the eyes, it will be unnatural for a human face, so the range within which the mouth can actually move within the face is set as the movable range of the feature point of the mouth. When the image processing device 100 uses BlendShape, the movable range of each of the multiple feature points may be set so as not to exceed the maximum value that can be expressed by the Key value of the shape key.

[0053] The avatar generation unit 118 may generate an avatar by further using the movable ranges of each of the plurality of feature points stored in the range storage unit 122. The avatar generation unit 118 may generate an avatar in which each part changes by moving each of the plurality of feature points for which the weighted feature amount is calculated by the second calculation unit 116 within the movable ranges stored in the range storage unit 122 according to a moving speed to which a weight value is applied. This makes it possible to prevent the face of the user 40 from becoming unnatural by emphasizing a part or the whole of the face of the user 40.

[0054] The display control unit 124 performs control so as to superimpose and display the avatar generated by the avatar generation unit 118 on the face portion of the imaging data of the user 40. The display control unit 124 may perform control so as to generate a display image in which the avatar generated by the avatar generation unit 118 is placed on the face portion of the imaging data of the user 40, and display the display image.

[0055] 4 is an explanatory diagram for explaining eye emphasis. Avatar 400 shows a state in which the eyes are not emphasized, and avatar 402 shows a state in which the eyes are emphasized.

[0056] When user 40 has a facial expression similar to avatar 400, in normal mode, avatar 400 is displayed as is, and in eye emphasis mode, avatar 402 is displayed.

[0057] In this way, by making it possible to emphasize the eyes of an avatar, it is possible to strongly communicate one's intentions to the other person and provide the other person with an interesting experience.

[0058] 5 is an explanatory diagram for explaining facial expression emphasis. Avatar 300 shows a state in which the facial expression is not emphasized, avatar 302 shows a state in which the facial expression is weakly emphasized, and avatar 304 shows a state in which the facial expression is strongly emphasized.

[0059] When user 40 has a facial expression like avatar 300, in normal mode, the user is displayed as avatar 300, in facial expression emphasis mode, the user is displayed as avatar 302 if the facial expression emphasis mode is set with a low weighting value, and in facial expression emphasis mode, the user is displayed as avatar 304 if the facial expression emphasis mode is set with a high weighting value.

[0060] In this way, by making it possible to emphasize the facial expression of the avatar, it is possible to give a good impression to the other person. Also, by making the weighting value changeable, it is possible to change the degree of emphasis of the facial expression as desired by the user 40.

[0061] 6 shows an example of a flow of processing by the image processing device 100. Here, an example will be described in which a normal mode, an eye emphasis mode, and an expression emphasis mode are prepared.

[0062] In S102, the first calculation section 114 analyzes the imaging data acquired by the imaging data acquisition section 112, and calculates the feature amount of each of a plurality of feature points on the face of the user 40.

[0063] In S104, the second calculation unit 116 determines whether or not the eye emphasis mode is set. If the eye emphasis mode is set, the process proceeds to S106, and if not, the process proceeds to S108. In S106, the second calculation unit 116 calculates weighted feature amounts by applying weight values ​​corresponding to the eyes to the feature amounts of the multiple feature points corresponding to the eyes of the user 40.

[0064] In S108, the second calculation unit 116 determines whether or not the facial expression emphasis mode is set. If the facial expression emphasis mode is set, the process proceeds to S110, and if not, the process proceeds to S116. In S110, the second calculation unit 116 calculates weighted feature amounts by applying weighting values ​​corresponding to each part to the feature amounts of a plurality of feature points corresponding to the eyes, eyebrows, cheeks, and mouth of the user 40.

[0065] In S112, the avatar generation unit 118 determines whether the destination of a feature point for which weighted feature amounts were calculated in S106 or S110 will exceed the movable range when the weighted values ​​are applied. If there is a feature point whose destination exceeds the movable range, the process proceeds to S114. In S114, the avatar generation unit 118 changes the destination of the feature point whose destination exceeds the movable range to within the movable range. For example, the avatar generation unit 118 changes the destination of the feature point whose destination exceeds the movable range to an intersection of a straight line connecting the position before movement and the position of the destination and an end of the movable range.

[0066] In S116, the avatar generation unit 118 generates an avatar. If the result of the determination in S108 is NO, that is, if the normal mode is set, the second calculation unit 116 generates an avatar based on the imaging data. For example, the avatar generation unit 118 generates a 3D model of the face of the user 40 using a depth information image included in the imaging data, generates texture data of the face of the user 40 using a captured image included in the imaging data, and generates the avatar by performing a process of pasting the texture data onto the 3D model.

[0067] In the eye emphasis mode, the avatar generation unit 118 generates an avatar using the weighted feature amounts of the multiple feature points corresponding to the eyes of the user 40 calculated in S106. In the facial expression emphasis mode, the avatar generation unit 118 generates an avatar using the weighted feature amounts of the multiple feature points corresponding to the eyes, eyebrows, cheeks, and mouth of the user 40 calculated in S110.

[0068] In S118, the display control unit 124 generates a display image in which the avatar generated in S116 is placed in the area of ​​the face of the user 40 in the imaging data, and performs control so as to display the display image.

[0069] If the avatar generation process is to be ended due to an instruction from user 40 or the like (YES in S120), the process ends, and if not (NO in S120), the process returns to S102 and continues.

[0070] 7 is a schematic diagram showing an example of a hardware configuration of a computer 1200 functioning as the image processing device 100 or the communication device 200. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of the device according to the present embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the device according to the present embodiment, and / or cause the computer 1200 to execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0071] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive, a solid state drive, etc. The computer 1200 also includes a legacy input / output unit such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0072] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0073] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0074] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0075] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.

[0076] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0077] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0078] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0079] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0080] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0081] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0082] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0083] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0084] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0085] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order. [Explanation of symbols]

[0086] 10 avatar display system, 20 network, 40 user, 100 image processing device, 102 imaging unit, 112 imaging data acquisition unit, 114 first calculation unit, 116 second calculation unit, 118 avatar generation unit, 120 mode management unit, 122 range storage unit, 124 display control unit, 200 communication device, 202 imaging unit, 300, 302, 304, 400, 402 avatar, 1200 computer, 1210 host controller, 1212 CPU, 1214 RAM, 1216 graphic controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1230 ROM, 1240 input / output chip

Claims

1. a first calculation unit that analyzes imaging data of a user's face and calculates feature amounts including a moving direction and a moving speed of each of a plurality of feature points of the user's face; a second calculation unit that calculates a weighted feature amount by applying a weighting value corresponding to a part of the face to each of the feature amounts of at least a part of the feature points of the user's face; an avatar generation unit that generates an avatar that moves in accordance with a facial movement of the user, the avatar generation unit generating the avatar based on the weighted feature amounts of each of the plurality of feature points; a display control unit that controls the avatar generated by the avatar generation unit to be displayed so as to be superimposed on a face portion of the user in the imaging data; An image processing device comprising:

2. the second calculation unit calculates the weighted feature amount by applying a weighting value corresponding to a part of the face to the moving speed included in the feature amount of each of the at least some of the plurality of feature points; The image processing device according to claim 1 , wherein the avatar generation unit generates the avatar in which each part changes by moving the at least some of the plurality of feature points according to the moving speed to which the weight value is applied.

3. a range storage unit that stores a movable range of each of the plurality of feature points of the user's face; Equipped with The image processing device according to claim 2 , wherein the avatar generation unit generates the avatar in which each part changes by moving, within the movable range, at the moving speed to which the weighted value is applied, for each of the at least some of the plurality of feature points.

4. A mode management unit that switches between a normal mode and an eye emphasis mode that emphasizes the user's eyes. Equipped with When the normal mode is set, the avatar generation unit generates the avatar based on the imaging data, 4. The image processing device according to claim 1, wherein when the eye emphasis mode is set, the second calculation unit calculates the weighted feature amounts by applying weight values ​​corresponding to the eyes to the feature amounts of each of a plurality of feature points corresponding to the user's eyes among the plurality of feature points of the user's face, and the avatar generation unit generates the avatar using the weighted feature amounts of the plurality of feature points corresponding to the user's eyes.

5. the mode management unit switches between the normal mode, the eye emphasis mode, and a facial expression emphasis mode that emphasizes a facial expression of the user; 5. The image processing device according to claim 4, wherein when the facial expression emphasis mode is set, the second calculation unit calculates weighted feature amounts by applying weighting values ​​corresponding to parts of the face to the feature amounts of each of a plurality of feature points corresponding to the user's eyes, eyebrows, cheeks, and mouth among the plurality of feature points of the user's face, and the avatar generation unit generates the avatar using the weighted feature amounts of the plurality of feature points corresponding to the user's eyes, eyebrows, cheeks, and mouth.

6. A mode management unit that switches between a normal mode and an expression enhancement mode that enhances the user's expression. Equipped with When the normal mode is set, the avatar generation unit generates the avatar based on the imaging data, 4. The image processing device according to claim 1, wherein when the facial expression emphasis mode is set, the second calculation unit calculates weighted features by applying weights corresponding to parts of the face to the feature amounts of each of a plurality of feature points corresponding to the user's eyes, eyebrows, cheeks, and mouth among the plurality of feature points of the user's face, and the avatar generation unit generates the avatar based on the weighted features of the plurality of feature points corresponding to the user's eyes, eyebrows, cheeks, and mouth and the feature amounts of a plurality of feature points other than the user's eyes, eyebrows, cheeks, and mouth.

7. A program for causing a computer to function as the image processing device according to any one of claims 1 to 3.

8. 1. A computer-implemented image processing method, comprising the steps of: a first calculation step of analyzing image data of a user's face and calculating feature amounts including a moving direction and a moving speed of each of a plurality of feature points of the user's face; a second calculation step of calculating weighted feature amounts by applying weighting values ​​corresponding to parts of the face to the feature amounts of at least a part of the feature points of the user's face; an avatar generation unit that generates an avatar that moves in accordance with a facial movement of the user, the avatar being generated based on the weighted feature amounts of each of the plurality of feature points; a display control step of controlling the avatar generated by the avatar generation unit to be displayed so as to be superimposed on a face portion of the user in the imaging data; An image processing method comprising:

Images