Imaging apparatus, control method, and program

Abstract

To provide an imaging apparatus, a control method, and a program allowing a photographer to practice portrait photographing in a desired photographing environment even without an actual model to be a subject, and improve the knowledge and skill of portrait photographing through the practice.SOLUTION: An imaging apparatus has a display unit that displays an image of an avatar superimposed on a live view screen, and the imaging apparatus acquires a user-side element including predetermined information on a user and the surroundings of the user, sets, according to a user operation, an avatar-side element including an element of the avatar changing through communication with the user and an element of the avatar not changing through the communication, performs analysis on the basis of the user-side element and the avatar-side element, and performs control to reflect a result of the analysis on the live view screen and the image of the avatar displayed superimposed on the live view screen.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an imaging device, a control method, and a program, and more particularly to an imaging device, a control method, and a program that assist a photographer in practicing portrait photography. [Background technology]

[0002] Photographing people using cameras (hereinafter referred to as "portraits") has been practiced for a long time. In recent years, portrait photography itself has become common and popular, as typified by the portrait mode of smartphones, and even beginners who do not have expensive equipment can easily take portraits.

[0003] While beginners can now easily take portraits, creating high-quality portraits requires advanced knowledge and skills. For example, in Patent Document 1, a background image is projected onto a plain background material behind the subject to support the photographer and create an appropriate atmosphere for photographing the subject, so that the photographer can draw out a favorable expression from the subject. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-217516 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 is only intended to support the photographer when taking a portrait of a person who is actually a model subject, and is therefore based on the premise that the photographer has prepared a person who will be the subject.

[0006] However, it is difficult for a photographer to always arrange for a model to be available in an environment desired by the photographer for portrait photography practice.

[0007] To provide an imaging device, a control method, and a program that enable a photographer to practice portrait photography in a desired shooting environment even if there is no actual model to be the subject, and that improves knowledge and skills in portrait photography through the practice. [Means for solving the problem]

[0008] In order to solve the above problems, the imaging device of the present invention is an imaging device having a display unit that superimposes an image of a virtual three-dimensional avatar on a live view screen, and is characterized in that it comprises: an acquisition means for acquiring a first element including information about a user and his / her surroundings; a setting means for setting a second element including elements of the three-dimensional avatar that change in communication with the user and elements of the three-dimensional avatar that remain unchanged in the communication in accordance with user operation; an analysis means for performing analysis based on the first element acquired by the acquisition means and the second element set by the setting means; and a control means for controlling the result of the analysis to be reflected in the live view screen and the image of the three-dimensional avatar superimposed thereon. Effect of the Invention

[0009] According to the present invention, even if there is no actual model to be the subject, a photographer can practice taking portraits in a desired shooting environment, and through this practice, he or she can improve knowledge and skills in portrait photography. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a functional configuration of an imaging device according to a first embodiment. [Diagram 2] 5A to 5C are diagrams for explaining an analysis process according to the first embodiment that is executed by the imaging device. [Diagram 3]4 is a flowchart of a photographing process according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of factors that affect a communication score according to the first embodiment. [Diagram 5] FIG. 4 is a diagram showing an example of a live view screen displayed on the display unit in FIG. 1 during the processing of steps S305 to S307 in FIG. [Figure 6] 11A to 11C are diagrams illustrating examples of how an avatar appears on a live view screen when the depth of field and the focal distance are changed. [Figure 7] 11 is a diagram for explaining an analysis process according to a second embodiment executed by the imaging device. FIG. [Figure 8] 1 is a conceptual diagram illustrating a method for estimating environmental light from the luminance distribution of a real object. [Figure 9] FIG. 8 is a diagram for explaining how an avatar appears when the lighting by the virtual light set in step S701 in FIG. 7 is reflected on the avatar on the live view screen. [Figure 10] FIG. 8 is a diagram for explaining how an avatar appears when the lighting due to the ambient light around the user estimated in step S703 in FIG. 7 is reflected on the avatar on the live view screen. [Figure 11] FIG. 13 is a diagram showing an example of a joint that should not be cut when taking a portrait. [Figure 12] 13 is a diagram for explaining an analysis process according to a third embodiment executed by the imaging device. FIG. [Figure 13] FIG. 13 is a diagram illustrating an example of 3D modeling of an avatar's wrist using a point cloud in CG. [Figure 14] 14 is a conceptual diagram showing the position of the wrist of the avatar shown in FIG. 13 in a virtual three-dimensional space and the imaging range during imaging by an imaging device associated with the virtual three-dimensional space. [Figure 15] FIG. 11 is a flowchart showing a joint disengagement determination process according to the third embodiment. [Figure 16] 16A to 16C are diagrams for explaining live view screens before and after a notification of disarticulation in step S1506 in FIG. 15 is displayed. [Figure 17] FIG. 13 is a diagram for explaining an analysis process according to a fourth embodiment executed by the imaging device. [Figure 18] FIG. 13 is a diagram illustrating an example of the relationship between an AI model that evaluates a captured image and its input parameters in accordance with the fourth embodiment. [Figure 19] FIG. 13 is a diagram showing an example of evaluation information of a captured image according to the fourth embodiment, which is generated by an analysis unit and displayed on a display unit. [Figure 20] FIG. 13 is a diagram showing an example of comprehensive information on a plurality of captured images according to the fourth embodiment, which is generated by an analysis unit and displayed on a display unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the embodiments described below.

[0012] In the imaging device according to the present invention, when a user starts practicing portrait photography, a virtual three-dimensional avatar (hereinafter, referred to as "avatar") is displayed on a live view screen of a display unit. That is, the imaging device according to the present invention provides a user with a VR (Virtual Reality) or AR (Augmented Reality) experience as if the avatar displayed on the display unit were present around the user.

[0013] (Embodiment 1) (Communication evaluation value) <Functional configuration of the imaging device> First, the functional configuration of an image capturing apparatus 100 according to the first embodiment will be described with reference to the block diagram of FIG.

[0014] As shown in FIG. 1, the imaging device 100 includes an information acquisition unit 110 that mainly acquires specified information, a processing unit 120 that performs specified processing within the imaging device 100, an operation unit 130 used to operate the imaging device 100, and an output unit 140 that outputs the specified information.

[0015] The operation unit 130 is used by the user to make settings related to shooting and other necessary settings and to input data. For example, the operation unit 130 is composed of a cross key, a stick, buttons, a touch panel display, and a microphone.

[0016] The information acquisition unit 110 includes an image sensor unit 111 , a voice acquisition unit 112 , a distance sensor unit 113 , and a setting unit 114 .

[0017] The image sensor unit 111 is an imaging element configured with an image sensor such as a CCD or CMOS that converts an optical image into an electrical signal (photographed image).

[0018] The voice acquisition unit 112 is a microphone that acquires voice uttered by the user as conversation information. The image capture device 100 may accept operations by voice, in which case the operation unit 130 inputs the voice.

[0019] The distance sensor unit 113 is a sensor capable of measuring the distance from the imaging device 100 to an arbitrary object and acquiring the distance information. Examples of this sensor include an RGB camera, an infrared camera, a stereo camera, a LiDAR (Light Detection and Ranging) sensor, and a millimeter wave sensor. Note that the distance from the imaging device 100 to an arbitrary object can also be measured using an image sensor, in which case the image sensor unit 111 may function as the distance sensor unit 113.

[0020] The setting unit 114 sets predetermined parameters for the imaging device 100. The predetermined parameters include F-number (aperture value, depth of field), ISO sensitivity, shutter speed, white balance, focal length, etc., and also include avatar parameters. The avatar parameters include, for example, age, sex, appearance, personality, facial expression, pose, position, etc. The settings for this setting unit 114 are input by operating the operation unit 130. The avatar settings will be described in detail later.

[0021] The processing unit 120 includes an image processing unit 121, an analysis unit 122, an avatar control unit 123, and a display control unit 124. The processing unit 120 is configured with a CPU, an MPU, a GPU, and the like.

[0022] The image processing unit 121 performs various image processing on the captured image input from the image sensor unit 111 in accordance with the parameters set by the setting unit 114 .

[0023] The analysis unit 122 performs analysis based on the conversation information acquired by the voice acquisition unit 112 , the distance information acquired by the distance sensor unit 113 , and the parameters set by the setting unit 114 .

[0024] The avatar control unit 123 generates an image and a voice of the avatar based on the avatar parameters set by the setting unit 114, and controls the behavior of the avatar. For example, if the avatar is set to have a bright and lively personality, the output unit 140 is controlled so that the avatar will speak spontaneously and be expressive. The speech of the avatar produced by this control is integrated with the user's voice acquired by the voice acquisition unit 112, and used as conversation information to be analyzed by the analysis unit 122.

[0025] The display control unit 124 integrates the captured image from the image sensor unit 111, the image processing performed on the captured image by the image processing unit 121, the analysis result by the analysis unit 122, and the behavior of the avatar controlled by the avatar control unit 123. After that, the display control unit 124 outputs the integrated result to the recording unit 141 and the display unit 143 of the output unit 140, which will be described later.

[0026] The output unit 140 includes a recording unit 141 , a sound output unit 142 , and a display unit 143 .

[0027] The recording unit 141 is composed of a recording medium, and records the captured image input from the image sensor unit 111. Examples of the recording medium that can be used include an optical disk, a magnetic disk, a hard disk, and a memory. At this time, the image and sound of the avatar are also recorded.

[0028] Sound output unit 142 outputs the speech of an avatar controlled by avatar control unit 123. Sound output unit 142 is configured by, for example, a speaker, headphones, or the like.

[0029] The display unit 143 displays a user's operation screen, a captured image, setting information, an avatar, analysis information, etc. The display unit 143 is configured, for example, with a liquid crystal display, an organic electroluminescence (organic EL) display, etc. Furthermore, the display unit 143 may display information and at the same time accept touch operations, in which case it is included as part of the operation unit 130.

[0030] <Overview of the imaging system> Next, the analysis process according to the first embodiment executed by the imaging device 100 will be described with reference to FIG.

[0031] In step S201, the setting unit 114 (setting means) performs settings on the imaging device 100. In this step, information on the virtual world, such as the appearance and pose of an avatar, is mainly acquired.

[0032] In step S202, information on the user and his / her surroundings is acquired by the image sensor unit 111, the voice acquisition unit 112, and the distance sensor unit 113 (acquisition means). In this step, mainly real-world information such as the user's voice and the distance between the user and the avatar is acquired.

[0033] In step S203, the analysis unit 122 (analysis means) analyzes the communication between the user and the avatar based on the information acquired in steps S201 and S202, and generates analysis information.

[0034] In step S204, the avatar control unit 123 and the display control unit 124 (control means) control the display unit 143 and the sound output unit 142 so as to reflect the contents of the analysis information generated in step S203 in the live view screen and the avatar superimposed thereon.

[0035] Each step will be described in detail below.

[0036] First, step S201 will be described in detail.

[0037] In step S201, the setting unit 114 sets the avatar's age, appearance, personality, facial expression, pose, position, etc. in response to user operations on the operation unit 130. The avatar is generated by the avatar control unit 123 based on the set avatar information, and the avatar behaves autonomously according to the settings. "Autonomous behavior according to the settings" refers to the avatar's actions, such as spontaneously talking a lot and being expressive when the personality is set to be bright and lively.

[0038] Here, the photographing process according to this embodiment, in which an avatar is displayed superimposed on a live view screen and photographed, will be described with reference to the flowchart of FIG.

[0039] First, parameters such as the sex, age, appearance, and personality of an avatar are set by the setting unit 114 in response to a user's operation on the operation unit 130 (step S301). At this time, the preset facial expressions and poses of the avatar are also read out.

[0040] Next, avatar control unit 123 generates an avatar based on the avatar parameters set in step S301 (step S302).

[0041] Next, in order to superimpose the avatar generated in step S302 onto the real world, the distance sensor unit 113 and the analysis unit 122 map the environment around the user (step S303). At this time, for example, a SLAM (Simultaneous Localization and Mapping) technique is used for mapping. The SLAM technique is well known, so a description thereof will be omitted.

[0042] Next, the initial position of the avatar is set according to the result of mapping the surrounding environment obtained in step S303 and the user's selection operation of the position of the avatar relative to the operation unit 130 or the display unit 143 (step S304).

[0043] This completes the initial setting of the avatar, and shooting begins with the avatar superimposed on display unit 143 (step S305). During shooting, the avatar is controlled to behave autonomously according to the parameters set in step S301. This control of the avatar is performed by avatar control unit 123.

[0044] During shooting, the avatar's facial expression, pose, and position are changed in response to user instructions. These user instructions are given by operating the operation unit 130 or by the user's voice operation (step S306). The user's voice operation will be described later.

[0045] Next, the analysis unit 122 determines whether or not the image capture has been completed (step S307). For example, it determines whether or not the user has performed an operation to end image capture using the operation unit 130. At this time, if the operation to end image capture has not been performed (NO in step S307), the process proceeds to step S306, and image capture continues. On the other hand, if the operation to end image capture has been performed (YES in step S307), the captured image (a portrait image in which the avatar is superimposed on the real object) is recorded in the recording unit 141, and this process ends.

[0046] Next, step S202 will be described in detail.

[0047] In step S202, the voice acquisition unit 112 acquires the voice of the user, and the distance sensor unit 113 (e.g., acquisition of three-dimensional information by a LiDAR sensor) acquires the distance between the user and the avatar (acquisition means). When shooting begins in step S305, the user can have interactive communication through conversation with the avatar displayed on the display unit 143. Here, "conversation" is a concept that includes voice exchanges between the user and the avatar, as well as silent parts. In addition, "interactive communication" includes, for example, communication in which the avatar looks embarrassed when the user compliments the avatar, or the avatar poses according to instructions from the user.

[0048] Next, steps S203 and S204 will be described in detail.

[0049] In step S203, the analysis unit 122 analyzes the information about the user and the avatar obtained in steps S201 and S202.

[0050] Furthermore, in step S204, the content of the analysis information obtained in step S203 is output to the display unit 143 and the sound output unit 142 so as to be reflected on the live view screen and the avatar.

[0051] Steps S203 and S204 affect each other, so they will be explained together.

[0052] The items to be analyzed in step S203 are not particularly limited, and may be information about the voice extracted from information about the user and the avatar. For example, the contents and amount of speech the user makes to the avatar, the frequency of silences, the volume of the user's voice, and the speech rate, the number of times the conversation between the user and the avatar overlap, and the ratio of the user's speech time to the avatar's speech time (talk / listen ratio) are exemplified.

[0053] In step S204, if the analysis indicates that the user has given instructions to the avatar regarding facial expression, pose, position, etc., that information is sent to avatar control unit 123, and avatar control unit 123 performs control according to the contents of the instructions. For example, if the contents are "Raise your hand," avatar control unit 123 controls the avatar to raise its hand, and the avatar with its hand raised is displayed on display unit 143 via display control unit 124.

[0054] Even if the instruction is not related to facial expression, pose, position, etc., the facial expression, pose, and position of the avatar may be influenced according to what the user said. For example, if the analysis shows that the user "praised" the avatar, the avatar control unit 123 controls the avatar to smile. This allows for more natural communication between the user and the avatar.

[0055] Furthermore, in step S203, the analysis unit 122 analyzes the quality of communication between the user and the avatar, taking into consideration the above-mentioned voice analysis, the avatar settings, the distance between the user and the avatar acquired by the distance sensor unit 113, and the like. The analysis result is quantified and calculated as a communication score.

[0056] Avatar settings can affect the communication score. For example, if the avatar is set to have a quiet personality, the communication score will decrease if the user talks too much to the avatar. Also, if the avatar's age is set low (i.e. the avatar is set to be a child), the communication score will increase if the user speaks slowly and in simple terms to the avatar.

[0057] The distance between the user and the avatar may also affect the communication score: for example, if the user gets too close to the avatar, the communication score will decrease.

[0058] The calculated communication score is output and displayed on display unit 143 in step S204. This allows the user to monitor the communication score in real time and provide immediate feedback on how the user is communicating with their avatar. For example, if the communication score drops due to a period of silence, the user can try to improve communication by talking more.

[0059] Furthermore, the communication score itself may be used as an index when calculating the communication score, and further, the communication score may be reflected in the behavior of the avatar. For example, if the communication score is low, the analysis unit 122 further lowers the communication score when the user approaches the avatar. In this case, the avatar control unit 123 controls the avatar to have a troubled expression. On the other hand, if the communication score is high, the analysis unit 122 maintains the communication score without lowering it even if the user approaches the avatar. In this case, the avatar control unit 123 controls the avatar to maintain a smiling expression.

[0060] Here, we will use Figure 4 to explain how each of the factors that affect the communication score mentioned so far affect each other.

[0061] As shown in FIG. 4, there are three main factors that affect the communication score: user-side factors 31, avatar-side factors 32, and communication score 33.

[0062] The element 31 (first element) on the user side is the element acquired in step S202 described with reference to FIG.

[0063] The avatar-side elements 32 (second elements) are elements acquired in step S201, as described in FIG. 2. The avatar-side elements 32 can be divided into "unchanging elements 322" and "changing elements 321" in communication between the user and the avatar. For example, the facial expression of the avatar changes through communication with the user and is therefore a "changing element 321," while the age of the avatar is an "unchanging element 322" that does not change during communication with the user.

[0064] In addition, other than the user's element 31, the elements that affect the change in the communication score are the communication score 33 itself and the "element 321 that changes due to communication" among the avatar's elements 32.

[0065] As shown by the dashed arrows in FIG. 4, the "variable elements 321" in communication change under the influence of the user's elements 31, the "unchanging elements 322" in communication, and the communication score 33.

[0066] Furthermore, as indicated by the solid arrows in FIG. 4, the communication score 33 changes under the influence of the user's element 31, the avatar's element 32, and the communication score 33 itself.

[0067] The factors that affect the communication score are not limited to those shown in Figure 4 above. If there are other factors that the user needs to consider when communicating with the avatar, they can be added as appropriate and combined with other factors.

[0068] One possible method for calculating the communication score is to use AI (Artificial Intelligence), for example.

[0069] The AI ​​model used in this calculation method is trained, for example, as follows.

[0070] First, the voice of the professional photographer and the distance from the model are obtained as parameters in a time series when the professional photographer actually takes a portrait of a model. In addition, the gender, age, appearance, and personality of the model, as well as the changes in the model's facial expression, pose, and position over time during the portrait shoot are also obtained as parameters. In addition, the communication score during the portrait shoot is always set to 100 points, and this is also obtained as a parameter.

[0071] Next, when the various acquired parameters are input, the AI ​​model is trained so that the output value (communication score) will be 100 points. By conducting this type of training with various professional photographers and situations, it is possible to create an AI model that scores 100 points for the way professional photographers communicate when taking portraits.

[0072] In this case, the accuracy can be further improved by learning portrait photography data from non-professional photographers. For example, when an amateur photographer actually takes a portrait using a model, various parameters including a parameter that gives a communication score of 70 points can be acquired, and the AI ​​model can be additionally trained so that when these parameters are input, the output value becomes 70 points. Similarly, for portrait photography of a person who has never taken a photo with a camera other than a smartphone, the AI ​​model can be additionally trained so that the communication score of the input parameters and the output value are both 30 points.

[0073] Furthermore, the AI ​​model may be additionally trained by changing the input parameter communication score and output value, taking into account factors that affect the photographer's communication skills. For example, if the photographer has owned a camera for a long time or has taken a large number of photos, the communication score and output of the input parameters are set higher since the photographer is likely to have higher communication skills.

[0074] As a result, an AI model can be created that has been trained to output a higher value for people with higher communication skills in portrait photography. This AI model is created in advance and loaded into the analysis unit 122. As a result, the user-side elements 31, the avatar-side elements 32, and the pre-update communication score 33 acquired during shooting with the imaging device 100 are input to the AI ​​model, whereby the updated communication score can be output.

[0075] As described above, in the imaging device 100 according to the first embodiment, analytical information on a plurality of elements is reflected in the avatar's behavior (changing elements 321) and communication score. As a result, the user can practice communication during portrait photography without having to prepare a subject (a model to be photographed).

[0076] <Screen display> Next, a live view screen according to the first embodiment displayed on the display unit 143 during the processing of steps S305 to S307 in Fig. 3 will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing an example of this live view screen. Fig. 6 is a diagram showing an example of how an avatar appears on the live view screen when the depth of field and the focal distance are changed.

[0077] An image of a real object 501 such as a landscape or an object in the real world captured by the image sensor unit 111 is output to a live view screen 500 exemplified in FIG.

[0078] Further, under the control of the display control unit 124, the virtual avatar 502 set by the setting unit 114 is displayed superimposed on the physical object 501. At this time, since the position of the avatar 502 in the real world is determined by the setting unit 114 and the avatar control unit 123, the avatar 502 is displayed superimposed on the physical object 501 only when the avatar 502 is included within the angle of view of the imaging device 100. Since the distance between the user (imaging device 100) and the avatar 502 is also acquired by the distance sensor unit 113, the analysis unit 122 can determine the distance relationship in the depth direction between the physical object 501 and the avatar 502. Therefore, under the control of the display control unit 124, when the avatar 502 is located in front of the physical object 501, the avatar is displayed on the live view screen 500 (the physical object 501 is hidden by the avatar 502). On the other hand, when the avatar 502 is positioned behind the physical object 501, the physical object 501 is displayed on the live view screen 500 (the avatar 502 is hidden by the physical object 501).

[0079] Furthermore, the communication score 504 calculated by the analysis unit 122 is also superimposed and displayed on the live view screen 500. At this time, the communication score may be displayed directly as a numerical value, or may be displayed graphically taking into consideration visual ease (for example, as shown in FIG. 5, the degree to which the hatched area fills the heart mark), or may be displayed in combination of both. Note that, before communication between the user and avatar 502 begins, when the communication score cannot be calculated, an appropriate initial value (for example, 50) is set and displayed on the live view screen 500.

[0080] In the live view screens 600a and 600b shown in Fig. 6, the F-number has been changed by the setting unit 114, and the depth of field has been made shallower than that of the live view screen 500. The live view screen 600a in Fig. 6(a) is a live view screen when the focus is on a physical object 501 in the distance, and the live view screen 600b in Fig. 6(b) is a live view screen when the focus is on an avatar 502 in the foreground.

[0081] Normally, when the depth of field is shallow, the degree of blur increases as the distance from the focused position increases. That is, as shown in FIG. 6(a), when the focus is set on a distant view, a distant real object 501 (landscape, object, etc.) should be displayed sharply, and a nearby real object should be displayed significantly blurred. However, since the avatar 502 is not a real object but a virtual object generated within the imaging device 100, it is not optically affected by the lens. Therefore, in the live view screen 600a, the image processing unit 121 performs blurring processing on the avatar 502. This allows the avatar 502 to be displayed on the live view screen 600a as if it were a real object.

[0082] At this time, the amount of blurring applied to avatar 502 during blurring processing is controlled according to the distance from the focused position to the real-world position of avatar 502. In other words, for example, the closer the distance of avatar 502 from the focused position, the smaller the amount of blurring applied to avatar 502 during processing can be made, so that avatar 502 can be displayed closer to how it appears in reality.

[0083] Also, when focusing on a desired real object, MF (Manual Focus) or AF (Auto Focus) operation is considered. However, since the avatar 502 is not a real object but a virtual object, the avatar 502 cannot be focused on by AF. Meanwhile, the image capture device 100 has a distance sensor unit 113 that acquires the distance between the user and the avatar 502. Therefore, when an AF operation is performed on the position where the avatar 502 exists on the live view screen 600a (the position of the avatar 502 in the real world), the focus adjustment is automatically performed in the image capture device 100 according to the distance acquired by the distance sensor unit 113. That is, in this case, as shown in the live view screen 600b in FIG. 6(b), the avatar 502 is displayed as if it is in focus without blurring by the image processing unit 121. Meanwhile, the real object 501 in the distant view is displayed blurred due to the optical effect of the lens.

[0084] As described above, the camera applies blur to the virtual avatar, which is superimposed on the live view screen, depending on the distance from the focus point, and also enables AF control of the virtual avatar, allowing users to practice taking portraits that make use of the blur effect.

[0085] As described above, according to the first embodiment, a user can practice taking portrait pictures, including communicating with a subject, even if the user does not prepare a subject (a model to be photographed).

[0086] (Embodiment 2) (Reflecting real and virtual light on avatars) Hereinafter, an imaging device 100 according to a second embodiment will be described with reference to Fig. 7 to Fig. 10. This embodiment is characterized in that the ambient light around the user and the virtual light set by the user are reflected in the lighting of the avatar on the live view screen. Note that in this embodiment, a description of parts that overlap with the first embodiment will be omitted.

[0087] First, the analysis process according to the second embodiment executed by the imaging device 100 will be described with reference to FIG.

[0088] In step S701, the same settings as in step S201 are performed, and the user can set at least one of the type, light intensity, color, position, and number of virtual light sources in the setting unit 114 through the operation unit 130. For example, if the color of the virtual light source is set to "warm sunlight," the color of the ambient light reflected on the avatar is changed to a warm color. If the type of virtual light source is a strobe light, the position and number can be set to "two white strobe lights diagonally behind the subject," "white strobe light from the camera reflected on the ceiling," "one light reflected from the strobe light umbrella," and the like. Here, the "virtual light source" is a light source that does not exist in reality (around the user) and is virtually installed within the imaging device 100, and the lighting state of the avatar changes depending on the setting of the virtual light source. In addition, in order to make the setting easier, "virtual light source" modes such as "backlight mode," "frontlight mode," and "side light mode" may be prepared and selected by the user. For example, when the "backlight mode" is selected, lighting in which sunlight is irradiated from behind the avatar is automatically set. That is, even if the real ambient light is direct light, the lighting reflected on the avatar looks as if it were backlit. Also, if the user does not set a virtual light source, the lighting from the real ambient light is reflected on the avatar. On the other hand, if the type of virtual light source is a strobe light, the lighting from both the virtual light source and the real ambient light is reflected on the avatar.

[0089] In step S704, if virtual light has been set by the user in step S701, image processing unit 121 performs lighting processing to perform virtual lighting on the avatar based on the information of the set virtual light. Thereafter, display control unit 124 performs control so that the image of the avatar after the lighting processing is displayed on display unit 143 in a superimposed manner on the live view screen.

[0090] Virtual light settings and lighting processing can be achieved, for example, by using 3D graphics functions. 3D graphics functions can place virtual objects and light sources in a virtual space within a computer and simulate their effects through calculations. In addition, the type, light intensity, color, position, number, etc. of the source of virtual light (hereinafter referred to as virtual light source) can be selected.

[0091] Since image processing unit 121 has the above-mentioned 3D graphics function, it can calculate virtual lighting for an avatar by importing an avatar generated by avatar control unit 123. Here, the user selects the virtual light source, type, light intensity, color, position, number, etc. through operation unit 130.

[0092] In the imaging device 100 according to the second embodiment, the image sensor unit 111 acquires the ambient light around the user (step S702), and the analysis unit 122 estimates the ambient light (step S703). Here, the items to be estimated are at least one of the type, light amount, color, position, and number of the light source (hereinafter, real light source) of the ambient light.

[0093] A representative method for estimating the environmental light is, for example, a method of estimating it from the luminance distribution formed on the surface of a physical object.

[0094] In a method of estimation based on the luminance distribution formed on the surface of a physical object, the strength, direction (position), type, and number of real light sources can be estimated by obtaining the normal and luminance of the physical object from an image captured by the imaging device 100.

[0095] Here, in order to obtain the normal of a physical object, the shape of the physical object needs to be known, but in this embodiment, this can be acquired by the distance sensor unit 113 when mapping the surrounding environment in step S303.

[0096] The method of estimating the ambient light from the luminance distribution of a physical object will be described in more detail with reference to Fig. 8. Here, as an example, the luminance distribution of a specific region P of a cubic physical object O shown in Fig. 8(a) is observed. Figs. 8(b) to (d) are diagrams showing an example of the luminance distribution of the specific region P when the physical object O is viewed from the +y direction.

[0097] For example, if a specific region P has a luminance distribution as shown in FIG. 8(b), it can be estimated that weak diffused light from the -x direction is irradiated as environmental light. If the luminance distribution is as shown in FIG. 8(c), it can be estimated that strong linear light is irradiated only on the left half from the +y direction. If the luminance distribution is as shown in FIG. 8(d), it can be estimated that there are two weak diffused lights in the -x and +x directions. Note that, if the image sensor unit 111 is an RGB sensor, the luminance value of each of RGB can be obtained, so that the color of the light can also be estimated.

[0098] For simplicity of explanation, the estimation of the ambient light irradiated on one specific area has been described, but the accuracy of the estimation can be improved by using multiple areas for the estimation.

[0099] In addition to the above methods, there are many other known methods for estimating ambient light, such as a method of estimating based on shadows formed by real objects. Since these are known methods, detailed explanations will be omitted, but the method of estimating ambient light used in this embodiment may be any method capable of estimating at least one of the type, light amount, color, position, and number of real light sources, or may be a combination of these methods.

[0100] If the type of virtual light source is set to strobe light in step S701 or if a virtual light source is not set, the ambient light is estimated in step S703, and a virtual light source that matches the real light source is set by image processing unit 121 and lighting processing is performed (step S704). The process of reflecting the virtual light source in the avatar has already been explained, so a description thereof will be omitted.

[0101] We have explained how the real-world ambient light around the user and the virtual light set by the user are reflected on the avatar, but these can also be used in combination. For example, by going out at night in real life and setting a strobe as a virtual light source to light the avatar, you can practice "taking portraits of night scenes using a strobe" without having to carry around any equipment.

[0102] <Screen display> Next, a live view screen according to the second embodiment displayed on the display unit 143 during the processing of steps S305 to S307 in Fig. 3 will be described with reference to Fig. 9 and Fig. 10. Fig. 9 is a diagram for explaining how an avatar looks when lighting based on the virtual light set in step S701 is reflected on the avatar on the live view screen. Fig. 10 is a diagram for explaining how an avatar looks when lighting based on the ambient light around the user estimated in step S703 is reflected on the avatar on the live view screen.

[0103] FIG. 9(a) is an example of a live view screen when no lighting processing is performed on the avatar, as in embodiment 1, and FIG. 9(b) is an example of a live view screen after the lighting using the virtual light set in step S701 has been reflected on the avatar.

[0104] As shown in Fig. 9(a), when no lighting processing is performed on the avatar, the avatar appears uniformly bright on the live view screen. In contrast, when a virtual light source (a single white strobe light) is set in front of the avatar, the live view screen shows a shadow in front of the avatar (the face is in shadow) due to virtual lighting, as shown in Fig. 9(b).

[0105] FIG. 10(a) is an example of a live view screen when no lighting processing is performed on the avatar, as in embodiment 1, and FIG. 10(b) is an example of a live view screen after the ambient lighting estimated in step S703 has been reflected on the avatar.

[0106] As shown in Fig. 10(a), when no lighting processing is performed on the avatar, the avatar appears uniformly bright on the live view screen even if a real light source (the sun) is present to the left of the avatar. In contrast, when a virtual light source that matches the real light source is set by estimating the ambient light, a corrected live view screen is obtained in which the left side of the avatar is bright and the right side is dark, with a shadow cast on the face, as shown in Fig. 10(b).

[0107] As described above, in the second embodiment, lighting is applied to the avatar on the live view screen that reflects the real-world ambient light around the user and the virtual light set by the user. This allows the user to take portraits while being aware of the effects of the light source, and allows the user to improve their knowledge and skills in taking pictures that make use of the light source. In addition, since there is no need to prepare equipment or an environment, even beginners in photography can easily practice.

[0108] (Embodiment 3) (Detection of joint breakage photos) Hereinafter, an imaging device 100 according to a third embodiment will be described with reference to Fig. 11 to Fig. 16. This embodiment is characterized in that it notifies the user that a photo the user has taken or is about to take is a photo cut off at an avatar's joint (joint cut photo). Note that in this embodiment, a description of parts that overlap with the first embodiment will be omitted.

[0109] Fig. 11 is a diagram showing an example of joints that should not be cut when taking a portrait. It is generally considered that a photograph with framing that cuts off joints shown in Fig. 11, such as the forehead 1101, neck 1102, shoulder 1103, wrist 1104, hip joint 1105, knee 1106, and ankle 1107, should be avoided in portrait photography.

[0110] First, the analysis process according to the third embodiment executed by the imaging device 100 will be described with reference to FIG.

[0111] In step S1203, the analysis unit 122 analyzes where the positions of the avatar's joints are located within the live view screen. The method of this analysis will be described below.

[0112] In general, an avatar is modeled by a process of connecting a plurality of points in a virtual three-dimensional space using CG (Computer Graphics) technology. In the third embodiment, the above process is performed by the avatar control unit 123. Furthermore, the point cloud used in modeling the avatar has coordinates in the virtual three-dimensional space. Therefore, the analysis unit 122 acquires the coordinate values ​​of a predetermined part of the avatar from the avatar control unit 123. Here, the predetermined part refers to, for example, the positions of joints such as the forehead, neck, shoulders, wrists, hip joints, knees, and ankles described above.

[0113] Next, the analysis unit 122 (coordinate correspondence means) maps the real space around the user using the distance sensor unit 113. After that, the real space around the user and the shooting range (angle of view) at the time of shooting by the imaging device 100 are associated with coordinates in a virtual three-dimensional space in which the avatar is modeled.

[0114] After that, the analysis unit 122 compares the coordinates of the predetermined part of the avatar with the coordinates of the boundaries between the inside and outside of the angle of view of the image capture device 100. This allows the analysis unit 122 to analyze where the positions of the avatar's joints are located within the live view screen. Furthermore, based on the analysis results, the analysis unit 122 can detect whether or not "the framing is such that the predetermined part of the avatar is cut off."

[0115] In step S1204, if the analysis unit 122 detects that a "predetermined part of the avatar is cut off in the framing," the detection result is displayed on the live view screen using the display control unit 124. This display may be performed when the user presses the shutter.

[0116] Fig. 13 is a diagram illustrating an example of 3D modeling of an avatar's wrist using a point cloud in CG. Each point in Fig. 13 has coordinates in a virtual 3D space, and by controlling these coordinates, the avatar control unit 123 can determine the location of each part of the avatar, not just the wrist. This allows the avatar control unit 123 to control the behavior (pose and position) of the avatar.

[0117] FIG. 14 is a conceptual diagram showing the position of the wrist of the avatar shown in FIG. 13 in a virtual three-dimensional space and the imaging range of imaging device 100 during imaging, which is associated with the coordinates of the virtual three-dimensional space.

[0118] 14 indicates the shooting range (angle of view) of the imaging device 100. Note that, although it is a square pyramid for convenience, the shooting range in the depth direction is not limited to the base of the pyramid, but includes infinity. In other words, the four side surfaces of the pyramid are the boundaries between the inside and outside of the angle of view.

[0119] In the third embodiment, during the processing of steps S305 to S307 in Fig. 3, the analysis unit 122 performs a process of determining whether or not the coordinates on any of the four side faces of the quadrangular pyramid shown in Fig. 14 match a predetermined part of the avatar (joint disarticulation determination process). If the coordinates match and the user is about to press the shutter, or has already pressed the shutter, the processing unit 120 notifies the user.

[0120] For the purpose of explaining the joint disarticulation determination process, the coordinates of point A forming the avatar's wrist joint shown in FIG. 14 are defined as (a, b, c), and the coordinates of point B, a part of the surface forming the boundary at the right end of the shooting range of imaging device 100, are defined as (x, y, z).

[0121] Next, the joint disengagement determination process according to this embodiment will be described with reference to the flowchart of FIG.

[0122] First, the timing for notifying the user that the camera is disengaged at a joint is set by the setting unit 114 in response to the user's operation on the operation unit 130 (step S1501). Here, the user can select the timing to be either before or after the shutter is released.

[0123] Next, the analysis unit 122 determines whether the avatar, which is the subject, is in focus (step S1502). Determining whether the subject is in focus is a basic function, and is widely installed in general cameras, so a description thereof will be omitted. If the subject (avatar) is in focus (YES in step S1502), the process proceeds to step S1503. Note that in this embodiment, the analysis unit 122 determines that "the user is about to release the shutter" when the subject is in focus, but this is not limiting. For example, when the user half-presses the release button (not shown) of the imaging device 100, the analysis unit 122 may determine that "the user is about to release the shutter" and proceed to step S1503 even if the subject (avatar) is not in focus.

[0124] Next, the analysis unit 122 determines whether the distance between point A forming the avatar's wrist joint and point B that is a part of the surface forming the boundary of the right end of the imaging range of the image capture device 100 is smaller than a predetermined distance R (step S1503). Here, the coordinates of point A are (a, b, c) and the coordinates of point B are (x, y, z), so the distance between points A and B is expressed by the following formula (1).

[0125]

number

[0126] If the distance between points A and B is less than a predetermined distance R (YES in step S1503), the analysis unit 122 detects that a joint break has occurred, and proceeds to step S1504. In this example, the analysis unit 122 determines that the avatar's wrist is cut at the right edge of the live view screen. On the other hand, if the distance between points A and B is R or more (NO in step S1503), the analysis unit 122 detects that a joint break has not occurred, ends this process, and continues shooting.

[0127] Next, it is determined whether the timing set in step S1501 is before the shutter is released (step S1504). If the timing set is before the shutter is released (YES in step S1504), the process proceeds to step S1506, a notice of disjoint is given to the user, and the process ends. On the other hand, if the timing set is after the shutter is released (NO in step S1504), no notice is given at this time, and the process proceeds to step S1505.

[0128] Next, it is determined whether or not the shutter has been released by the user fully pressing the release button (not shown) of the imaging device 100 (step S1505). If the shutter has been released (YES in step S1505), the process proceeds to step S1506, where a disjoint is notified to the user, and this process ends.

[0129] Here, the process of comparing the distance between point A forming the avatar's wrist joint and a part of point B on the surface forming the boundary of the right end of the imaging range of the imaging device 100 has been described as an example, but is not limited to this. That is, point A is applied to all of the joint parts of the avatar described above with reference to FIG. 11. Also, point B is applied to all of the boundaries of the imaging range of the imaging device 100. By doing so, the analysis unit 122 can detect "whether or not a specific part of the avatar is cut off in the framing." Note that the joint parts of a model that is a subject in the real world must be detected using deep learning or the like, and the processing load for detection is large. In contrast, in this embodiment, the coordinate information of all of the joint parts of the avatar described above with reference to FIG. 11 is stored in advance in the imaging device 100, and the detection process of the joint parts is not necessary. Therefore, without imposing a large processing load on the imaging device 100, the user can practice framing that does not cause joint cutoff in portrait photography.

[0130] So far, we have described a method for detecting disarticulation by comparing coordinates in three-dimensional space, but it is also possible to detect disarticulation by comparing two-dimensional coordinates after the three-dimensional space is projected onto a plane. In CG technology, in order to render a three-dimensional model on a two-dimensional plane such as a display, a three-dimensional to two-dimensional conversion is performed, and this process is called projection. Projection methods include parallel projection and perspective projection, which are well known and will not be described here, but it is also possible to compare the coordinates of the joints converted into two-dimensional coordinates by these methods with the coordinates of the boundary of the shooting range. This reduces the load of the disarticulation judgment process.

[0131] The avatar joints detected by the image capture device 100 according to the third embodiment are not limited to the above examples, and the user may add or remove any of the joints as appropriate.

[0132] <Screen display> Next, the live view screen before and after the notification of disarticulation in step S1506 in FIG. 15 is displayed will be described with reference to FIG.

[0133] As shown in FIG. 16(a), when a user has taken or is about to take a photo in which the avatar's joint (here, the neck) is cut, the analysis unit 122 detects that a joint cut has occurred in the joint cut determination process of FIG. 15, and notifies the user of the joint cut. In the third embodiment, the display control unit 124 displays a joint cut notification on the display unit 143. For example, as shown in FIG. 16(b), a message such as "The joint is cut" is displayed superimposed on the live view screen to notify the user of the joint cut, i.e., to notify the user that the framing is such that the joint is cut.

[0134] If you are a beginner photographer, you may continue shooting without noticing that the lens is cut at the joint, so being notified can help reduce missed shots.

[0135] It should be noted that the message format is not limited to that shown in FIG. 16(b) as long as it can notify the user of the joint break. For example, the avatar may notify the user of the joint break through communication. By notifying the user of the joint break by the avatar itself pointing it out or showing an angry face, it is expected that the effect of not impairing the user's immersive feeling when taking a picture can be expected.

[0136] As described above, in the third embodiment, when a user takes or is about to take a photo in which an avatar's joint is cut off, the user is notified of this fact. This allows the user to develop a sense for avoiding framing that is cut off by the subject's joints. As a result, it is possible to reduce missed shots when photographing real people.

[0137] (Embodiment 4) (Quality evaluation value of captured image) Hereinafter, an imaging device 100 according to a fourth embodiment will be described with reference to Fig. 17 to Fig. 20. This embodiment is characterized in that evaluation information of an image captured by the imaging device 100 is generated and provided to a user. Note that in this embodiment, a description of parts that overlap with the first embodiment will be omitted.

[0138] In the fourth embodiment, an image captured by the imaging device 100 is analyzed, and evaluation information is generated based on the analysis results and provided to the user. By referring to the evaluation information, the user can provide feedback to his / her own portrait photography, thereby improving his / her photography knowledge and skills.

[0139] First, the analysis process according to the fourth embodiment executed by the imaging device 100 will be described with reference to FIG.

[0140] In step S1703, the analysis unit 122 (evaluation generation means) analyzes the captured image and generates evaluation information. The captured image is a portrait image captured by the imaging device 100 through communication with an avatar, in which the avatar is superimposed on a real object, and is an image recorded in the recording unit 141 when an operation to end shooting is performed in step S307. The user can select whether or not to generate evaluation information for any captured image recorded in the recording unit 141 via the operation unit 130.

[0141] In step S1704, the evaluation information generated in step S1703 is displayed on the live view screen of the display unit 143 via the display control unit 124. In this way, the evaluation information is provided to the user.

[0142] Here, the items to be analyzed for the captured image include, for example, the brightness, contrast, saturation, hue, sharpness, composition, amount of blur, and noise of the captured image, as well as the facial expression, pose, and position of the avatar in the captured image, etc. Based on the analysis of these items, for example, the entire captured image or scores and graphs for each item are generated as evaluation information.

[0143] One possible method for calculating the score of a captured image is to use AI, for example.

[0144] The AI ​​model used in this calculation method is trained, for example, as follows.

[0145] First, a portrait photograph taken by a professional photographer is obtained, and various parameters are extracted from it.

[0146] Next, when the various acquired parameters are input, the AI ​​model is trained so that the output value (score of the captured image) will be 100 points. By conducting this type of training using portrait photo patterns taken by various professional photographers, it is possible to create an AI model that scores portrait photos taken by professional photographers at 100 points.

[0147] In addition, for things that are difficult to quantify, such as composition, facial expressions, and poses, an AI model can be trained to have an unspecified number of people rate each photo, and then the average score becomes the output value (score for the captured image).

[0148] Furthermore, if an AI model is trained so that when the photo itself is input, it outputs the average score given by the unspecified number of people, it is possible to create an AI model that outputs individual scores for facial expression, pose, composition, etc. In this case, each element of facial expression, pose, and composition is scored by the AI, and the output value becomes the input when calculating the evaluation value of the captured image.

[0149] Fig. 18 is a diagram illustrating the relationship between an AI model that evaluates a captured image and its input parameters according to the fourth embodiment. In Fig. 18, parameters that are easy to quantify, such as sharpness and noise, are directly extracted from the captured image, and the values ​​are input to the captured image evaluation AI model. On the other hand, for parameters that are difficult to quantify, such as composition, facial expression, and pose, a photo is input to each of the above-mentioned AI models, and the output values ​​from each of the AI ​​models are input to the captured image evaluation AI model. Note that there are various methods for quantifying sharpness, noise, etc., and since these are publicly known, their explanations will be omitted.

[0150] It is also possible to simply label expressions. For example, by assigning a numerical value to each facial expression, such as 1 for a smiling face and 2 for a sad face, and using this as input, the system can learn the tendencies of portrait photographs taken by professional photographers.

[0151] Accuracy can be further improved by having the AI ​​model learn portrait photos taken by non-professional photographers. For example, when various parameters of portrait photos taken by amateur photographers are used as input, the AI ​​model may be trained so that the output value (score of the captured image) is 70 points. Similarly, for portrait photos taken by people who have never taken photos with a camera other than a smartphone, the AI ​​model may be trained so that the output value (score of the captured image) is 30 points. Furthermore, for portrait photos that have won prizes in contests, etc., the AI ​​model may be trained by setting a high output value (score of the captured image).

[0152] As a result, an AI model can be created that has been trained to output a higher value for a higher quality portrait photograph. This AI model is created in advance and loaded into the analysis unit 122. This enables the analysis unit 122 to generate evaluation information for an image captured by a user using the imaging device 100, using the AI ​​model.

[0153] Fig. 19 is a diagram showing an example of evaluation information of a captured image according to the fourth embodiment, which is generated by the analysis unit 122 and displayed on the display unit 143. The evaluation information shown in Fig. 19 is displayed on the display unit 143 of the imaging device 100. As shown in Fig. 19, the evaluation information includes a selected captured image, an overall score, values ​​and graphs of each item used in the analysis, an overall comment, and the like.

[0154] In the example shown in Figure 19, it can be seen that the value of "Composition" is low. This allows the user to recognize points that should be reviewed when taking photos, such as "Is the composition monotonous?" or "Does the composition make the subject stand out?"

[0155] Also, items such as "contrast" and "saturation" were mentioned as parameters for calculating the evaluation value of a photographed image, but a high value for these items does not necessarily mean that the photograph is good; they are merely indicators of photographic expression. Therefore, it is possible to consider a way of displaying such values ​​to indicate a classification.

[0156] For example, in Figure 19, when contrast is high, the points are displayed as "sharp," and when contrast is low, the points are displayed as "soft." Similarly, when saturation is high, the points are displayed as "vivid," and when saturation is low, the points are displayed as "pale."

[0157] Furthermore, an overall comment based on the overall analysis may be generated and displayed. In the example shown in Fig. 19, since the value of "Expression" is high, the comment "The subject's facial expression is very lively!" is displayed. This allows the user to know his / her strengths in portrait photography, for example, "I'm good at bringing out the facial expressions of the subject."

[0158] Here, as an example, because the "Expression" category is high, the comment "The subject's facial expression is very lively!" is displayed, but just because the facial expression is not lively does not mean that the "Expression" category will be low. Specifically, even if the facial expression is not lively, it is possible to express it with an annui-like expression in a portrait. Therefore, the values ​​of "Expression" and other categories are also calculated by making a multifaceted judgment.

[0159] In the example shown in FIG. 19, the overall comments are positive, but are not limited to this and may be advice to the user, for example. Specifically, the comment may be "Get a little closer to the subject to make it stand out." This allows the user to know about shortcomings, such as "The theme of my photo is not clear."

[0160] Although the generation of evaluation information for one captured image has been described so far, information that integrates the generated evaluation information may be generated. Specifically, in the third embodiment, the analysis unit 122 may generate evaluation information for multiple captured images and generate information that integrates the multiple evaluation information (hereinafter, referred to as "integrated information"), rather than just generating evaluation information for one captured image. This allows the user to check the tendency of his / her captured images.

[0161] 20 is a diagram showing an example of comprehensive information of a plurality of captured images according to the fourth embodiment, which is generated by the analysis unit 122 and displayed on the display unit 143. The "number of samples" is the number of images for which evaluation information was generated and which were used when generating the comprehensive information.

[0162] The values ​​of each item, such as "facial expression" and "pose," are displayed as the average value among the samples. By knowing the average value of each item, users can more accurately understand their own strengths and weaknesses.

[0163] In addition, the evaluation information shown in FIG. 19 displays the degree of "clear" / "soft" and the degree of "pale" / "vivid" of the captured image, while the overall information shown in FIG. 20 displays the number of images broken down by each degree among the samples. Specifically, out of 100 samples, there are 2 images that are "clear", 80 images that are "slightly clear", 8 images that are "medium", 10 images that are "slightly soft", and 0 images that are "soft". This allows the user to understand a tendency, for example, that "I often take clear portraits". Based on the above tendency, a general comment such as "I'll try taking soft portraits as well" may be displayed. This allows the user to expand the range of expression in portrait photography.

[0164] In the fourth embodiment, the analysis items of the captured image are not limited to the elements listed above, and may be appropriately added if there are elements to be considered in evaluating portrait photography.

[0165] As described above, according to the fourth embodiment, the user can check the evaluation information and overall information of the image he / she has taken, and can grasp the strengths, weaknesses, and tendencies of his / her portrait photography. Furthermore, by feeding back the information to his / her photography, it becomes possible to improve his / her photography knowledge and skills and to broaden the scope of expression.

[0166] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various forms within the scope of the present invention are also included in the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined.

[0167] For example, by combining the second and fourth embodiments, it is possible to analyze and evaluate a captured image taking into account the lighting of the subject (avatar). Specifically, as shown in the live view screens of Fig. 9(b) and Fig. 10(b), if the lighting is not good and only the subject's face is dark, the overall score may be reduced.

[0168] In addition, as general information, a breakdown of the number of "front-lit," "side-lit," and "backlit" samples may be displayed. If the ratio of "backlit" samples is very small, the user can expand the scope of their photographic expression by suggesting, "Let's try taking more backlit portraits."

[0169] In the above embodiment, a case where a "person" is used as an avatar has been described as an example, but this is not limited to this example and an animal may be used. For example, since dogs and cats are in high demand as pets, the present invention can be used for practicing pet photography by using dogs and cats as avatars.

[0170] In addition, in each embodiment of the present invention, from the perspective of others, the photographer is speaking to the model (avatar) in various ways even though there is no model in front of the imaging device 100. For this reason, it is usually assumed that the imaging device 100 will be used in a space where the photographer does not care about the eyes of others, such as one's own room or a studio. However, in order to practice portrait photography, pet photography, and the like even in a place where the eyes of others are concerned, the photographer may be able to silently perform processing equivalent to speaking to the imaging device 100. In this case, the photographer may be able to register the contents of the voice in advance as text, or a predetermined voice content may be preset in the imaging device 100. In this way, when using the imaging device 100 in a place where the eyes of others are concerned, the photographer can silently perform processing equivalent to speaking to the model by selecting the voice content registered (or preset) as text displayed on the touch panel display of the operation unit 130.

[0171] In addition, in addition to the shooting mode (first shooting mode) in which an avatar is superimposed on the live view screen as described in embodiments 1 to 4, the imaging device 100 may also have a normal shooting mode (second shooting mode) in which an avatar is not superimposed on the live view screen.

[0172] (Other embodiments) In this embodiment, a program for implementing one or more functions may be provided to a computer of the system or device via a network or storage medium, and the system controller of the system or device may read and execute the program. The system controller may have one or more processors or circuits, and may include multiple separate system controllers or a network of multiple separate processors or circuits to read and execute the executable instructions.

[0173] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0174] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0175] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An imaging device having a display unit that superimposes an image of a virtual three-dimensional avatar on a live view screen, comprising: an acquisition means for acquiring a first element including information about a user and his / her surroundings; a setting means for setting a second element including elements of the three-dimensional avatar that change in communication with the user and elements of the three-dimensional avatar that remain unchanged in the communication in accordance with user operation; an analysis means for performing analysis based on the first element acquired by the acquisition means and the second element set by the setting means; and a control means for controlling the results of the analysis to be reflected in the live view screen and the image of the three-dimensional avatar superimposed thereon. (Configuration 2) The imaging device according to configuration 1, wherein the acquisition means acquires the user's voice as the first element. (Configuration 3) The imaging device according to configuration 1 or 2, wherein the acquisition means acquires a distance between the user and the three-dimensional avatar as the first element. (Configuration 4) The imaging device described in any one of configurations 1 to 3, characterized in that the acquisition means acquires at least one of the type, light intensity, color, position, and number of light sources of the ambient light around the user as the first element. (Configuration 5) An imaging device described in any one of configurations 1 to 4, characterized in that the setting means further sets at least one of the type, light intensity, color, position, and number of the light source of virtual light that provides virtual lighting for the three-dimensional avatar in response to user operation. (Configuration 6) An imaging device described in any one of configurations 1 to 5, characterized in that the elements of the three-dimensional avatar that do not change with the communication include at least one of the three-dimensional avatar's age, gender, appearance, and personality, and the elements of the three-dimensional avatar that change with the communication include at least one of the three-dimensional avatar's facial expression, pose, and position. (Configuration 7) An imaging device as described in configuration 2, further comprising a sound output unit that outputs the speech of the three-dimensional avatar, wherein the control means controls the information based on the analysis means to be reflected in the speech of the three-dimensional avatar that is output from the sound output unit. (Configuration 8) The imaging device described in Configuration 7, characterized in that the analysis means analyzes, based on the first and second elements, at least one of the content and amount of speech of the user to the three-dimensional avatar, the frequency of silence, the volume of the user's voice, and the speaking speed of the user, the number of overlapping conversations between the user and the three-dimensional avatar, and the ratio between the user's speaking time and the three-dimensional avatar's speaking time. (Configuration 9) An imaging device described in any one of configurations 1 to 8, characterized in that the analysis means analyzes the quality of communication between the user and the three-dimensional avatar based on the first and second elements, and generates a quantified analysis result. (Configuration 10) The imaging device according to configuration 9, wherein the analysis means changes the analysis result in accordance with the element of the three-dimensional avatar that changes through the communication, the first element, and the analysis result. (Configuration 11) The imaging device described in any one of configurations 1 to 10, further comprising a coordinate matching means for matching the angle of view of the imaging device to coordinates in a virtual three-dimensional space in which the three-dimensional avatar is modeled, wherein the analysis means analyzes whether a photo taken or is about to be taken by the user is a photo cut off by a joint of the three-dimensional avatar based on the position of the joints of the three-dimensional avatar in the virtual three-dimensional space and the angle of view of the imaging device matched to coordinates in the virtual three-dimensional space by the coordinate matching means. (Configuration 12) An imaging device as described in any one of configurations 1 to 11, further comprising a performance evaluation generation means for generating a performance evaluation for a captured image in which the three-dimensional avatar is superimposed on a real object, the performance evaluation being generated by analyzing at least one of the brightness, contrast, saturation, hue, sharpness, composition, amount of blur, and noise of the captured image, and at least one of the facial expression, pose, and position of the avatar in the captured image. (Configuration 13) The imaging device described in Configuration 12, wherein the performance evaluation generation means generates a performance evaluation for a plurality of captured images taken by the imaging device, and generates comprehensive information by combining all the generated evaluation information. (Configuration 14) An imaging device described in any one of configurations 1 to 13, characterized in that the control means performs a blurring process on the three-dimensional avatar superimposed on the live view screen in accordance with the depth of field and focusing distance determined by the imaging device. (Configuration 15) The imaging device described in Configuration 4, wherein the control means reflects at least one of the type, light intensity, color, position, and number of the light source of the ambient light acquired by the acquisition means in the lighting of the three-dimensional avatar superimposed on the live view screen. (Configuration 16) The imaging device described in Configuration 5, wherein the control means reflects at least one of the type, light intensity, color, position, and number of the light source of the virtual light, which are set in response to the user operation, in the lighting of the three-dimensional avatar superimposed and displayed on the live view screen. (Configuration 17) An imaging device described in any one of configurations 1 to 16, characterized in having a first shooting mode in which the three-dimensional avatar is superimposed on the live view screen, and a second shooting mode in which the three-dimensional avatar is not superimposed on the live view screen. (Method 1) A control method for an imaging device having a display unit that superimposes an image of a virtual three-dimensional avatar on a live view screen, the control method comprising: an acquisition step of acquiring a first element including information about a user and his / her surroundings; a setting step of setting a second element including elements of the three-dimensional avatar that change in communication with the user and elements of the three-dimensional avatar that do not change in the communication in accordance with user operation; an analysis step of performing an analysis based on the first element acquired in the acquisition step and the second element set in the setting step; and a control step of controlling so that the results of the analysis are reflected in the live view screen and the image of the three-dimensional avatar superimposed thereon. (Program 1) A program for causing a computer to function as each of the means of the imaging device described in any one of configurations 1 to 17. [Explanation of symbols]

[0176] 100 Imaging device 110 Information Acquisition Department 111 Image sensor section 112 Voice Acquisition Unit 113 Distance sensor unit 114 Setting section 120 Processing section 121 Image processing section 122 Analysis Department 123 Avatar Control Unit 124 Display control section 130 Operation section 140 Output section 141 Recording Department 142 Sound output section 143 Display section

Claims

1. An imaging device having a display unit that displays an image of a virtual three-dimensional avatar superimposed on a live view screen, An acquisition means for acquiring a first element including information about a user and the surroundings; a setting means for setting a second element including an element of the three-dimensional avatar that changes in response to communication with the user and an element of the three-dimensional avatar that does not change in response to the communication in response to a user operation; an analysis means for performing an analysis based on the first element acquired by the acquisition means and the second element set by the setting means; a control means for controlling the live view screen and the image of the three-dimensional avatar superimposed thereon so as to reflect the result of the analysis; An imaging device comprising:

2. The imaging device according to claim 1 , wherein the acquisition means acquires a voice of the user as the first element.

3. 2. The imaging device according to claim 1, wherein the acquisition means acquires a distance between the user and the three-dimensional avatar as the first element.

4. 2. The imaging device according to claim 1, wherein the acquisition means acquires at least one of the type, amount of light, color, position, and number of light sources of environmental light around the user as the first element.

5. The imaging device according to claim 1, characterized in that the setting means further sets at least one of the type, light intensity, color, position, and number of a light source of virtual light that provides virtual lighting for the three-dimensional avatar in response to user operation.

6. The imaging device described in claim 1, characterized in that the elements of the three-dimensional avatar that do not change with the communication include at least one of the three-dimensional avatar's age, gender, appearance, and personality, and the elements of the three-dimensional avatar that change with the communication include at least one of the three-dimensional avatar's facial expression, pose, and position.

7. A sound output unit that outputs a speech sound of the three-dimensional avatar, 3. The imaging device according to claim 2, wherein the control means controls so that the information based on the analysis means is reflected in the speech of the three-dimensional avatar outputted from the sound output unit.

8. The imaging device described in claim 7, characterized in that the analysis means analyzes, based on the first and second elements, at least one of the content and amount of speech of the user to the three-dimensional avatar, the frequency of silence, the volume of the user's voice, and the speaking speed of the user, the number of overlapping conversations between the user and the three-dimensional avatar, and the ratio of the user's speaking time to the three-dimensional avatar's speaking time.

9. The imaging device according to claim 1, characterized in that the analysis means analyzes the quality of communication between the user and the three-dimensional avatar based on the first and second elements, and generates a quantified analysis result.

10. 10. The imaging apparatus according to claim 9, wherein the analysis means changes the analysis result in accordance with the element of the three-dimensional avatar that changes due to the communication, the first element, and the analysis result.

11. a coordinate matching unit for matching an angle of view of the imaging device to coordinates in a virtual three-dimensional space in which the three-dimensional avatar is modeled; The imaging device described in claim 1, characterized in that the analysis means analyzes whether a photo taken or is about to be taken by the user is cut by a joint of the three-dimensional avatar based on the position of the joints of the three-dimensional avatar in the virtual three-dimensional space and the angle of view of the imaging device that has been associated with coordinates in the virtual three-dimensional space by the coordinate matching means.

12. a performance evaluation generating means for generating a performance evaluation of a captured image, in which the three-dimensional avatar is superimposed on a real object, captured by the imaging device; The imaging device described in claim 1, characterized in that the evaluation of the quality is generated by analyzing at least one of the brightness, contrast, saturation, hue, sharpness, composition, amount of blur, and noise of the captured image, and at least one of the facial expression, pose, and position of an avatar in the captured image.

13. The imaging device according to claim 12, characterized in that the performance evaluation generating means generates the performance evaluation for a plurality of captured images taken by the imaging device, and generates comprehensive information by combining all the generated evaluation information.

14. The imaging device according to claim 1, characterized in that the control means performs blurring processing on the three-dimensional avatar superimposed and displayed on the live view screen in accordance with the depth of field and focusing distance determined by the imaging device.

15. The imaging device according to claim 4, characterized in that the control means reflects at least one of the type, light intensity, color, position, and number of the light source of the ambient light acquired by the acquisition means in the lighting of the three-dimensional avatar superimposed on the live view screen.

16. The imaging device described in claim 5, characterized in that the control means reflects at least one of the type, light intensity, color, position, and number of the light source of the virtual light, which are set in response to the user operation, in the lighting of the three-dimensional avatar superimposed and displayed on the live view screen.

17. 2. The imaging device according to claim 1, further comprising a first shooting mode in which the three-dimensional avatar is superimposed on the live view screen, and a second shooting mode in which the three-dimensional avatar is not superimposed on the live view screen.

18. A method for controlling an imaging device having a display unit that displays an image of a virtual three-dimensional avatar superimposed on a live view screen, comprising: An acquisition step of acquiring a first element including information about a user and its surroundings; a setting step of setting a second element including an element of the three-dimensional avatar that changes in response to communication with the user and an element of the three-dimensional avatar that does not change in response to the communication in response to a user operation; an analysis step of performing an analysis based on the first element acquired in the acquisition step and the second element set in the setting step; a control step of controlling the live view screen and the image of the three-dimensional avatar superimposed thereon to reflect the result of the analysis; A control method comprising:

19. A program for causing a computer to function as each of the means of the imaging apparatus according to claim 1.

Images