Information processing device, information processing method, and program
By detecting the characteristics of the subject in the information processing device and changing the display mode of the 3D model, the problem that 3D objects in the prior art cannot respond to the observer's movement is solved, and the interactive movement and reaction of 3D objects are realized, which enhances the immersion feeling and a sense of existence.
Patent Information
- Application Number
- CN202080045553.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-06-28
- Filing Date
- 2020-06-19
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-06-19
AI Technical Summary
Existing 3D video technology cannot make the generated 3D objects respond to the observer's motion or state and cannot perform interactive motion or reaction.
An information processing device is designed, including a control unit and a detection unit, to change the display mode of the 3D model by detecting features of the subject in the captured image, such as the direction of sight and facial expressions, so that it can be executed relative to the observer. Interactive movement or reaction.
It realizes the interactive motion or reaction of 3D objects in capturing images, enhances immersion and presence, and while maintaining authenticity, it can quickly and efficiently change the direction and brightness of the 3D model, adapt to the movement of the subject and state.
Smart Images

Figure CN114008582B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable a 3D object to perform interactive motion or reaction in response to the motion of an observer. Background Art
[0002] A method has conventionally been proposed that generates a 3D object in a viewing space by using information obtained by sensing a real 3D space, for example, a multi-view video obtained by capturing images of a subject from different viewpoints, and generates a video (also called a volumetric video) in which the object appears as if the object exists in the viewing space (for example, Patent Document 1).
[0003] Reference List
[0004] Patent Literature
[0005] Patent Document 1: WO 2017 / 082076 A Summary of the invention
[0006] Technical issues
[0007] The video generated by the above method is a 3D video generated by using the actual video of the subject. Therefore, the observer who observes the 3D video can enjoy the sense of immersion and presence as if the subject is close to the viewer.
[0008] However, it is not possible to make the generated 3D object perform movement or reaction in response to the movement or state of the observer. That is, it is not possible for the 3D object to perform interactive movement or reaction relative to the observer.
[0009] In view of the above, the present disclosure proposes an information processing apparatus, an information processing method, and a program capable of causing a 3D object displayed in a captured image including a subject as an observer to perform interactive motion or reaction with respect to the subject.
[0010] Solution to the problem
[0011] In order to solve the above problems, an information processing device according to the content of the present disclosure is an information processing device as follows, which includes: a control unit, which controls the 3D model displayed in the captured image; and a detection unit, which detects the features of the subject displayed in the captured image at the same time as the 3D model, wherein the control unit changes the display mode of the 3D model according to the features of the subject detected by the detection unit. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 An overview of the process from capturing images to generating 3D models is shown.
[0013] Figure 2 This is an illustration of an AR application.
[0014] Figure 3 is a first diagram showing an example of an operation state of the mobile terminal according to the first embodiment.
[0015] Figure 4 is a second diagram showing an example of an operation state of the mobile terminal according to the first embodiment.
[0016] Figure 5 is a hardware block diagram showing an example of the hardware configuration of the mobile terminal according to the first embodiment.
[0017] Figure 6 is a functional block diagram showing an example of a functional configuration of the mobile terminal according to the first embodiment.
[0018] Figure 7 This is a diagram for explaining the method of measuring the line of sight direction.
[0019] Figure 8 A method of changing the drawing direction of a 3D model is shown.
[0020] Fig. 9 is a flowchart showing an example of the flow of a process performed by the mobile terminal according to the first embodiment.
[0021] Fig.10 An example of an operation state of the mobile terminal according to the second embodiment is shown.
[0022] Fig.11 is a flowchart showing an example of the flow of a process performed by the mobile terminal according to the second embodiment.
[0023] Fig.12 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to the third embodiment.
[0024] Fig.13 is a flowchart showing an example of the flow of a process performed by the mobile terminal according to the third embodiment.
[0025] Fig.14 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to the fourth embodiment.
[0026] Fig.15 is a flowchart showing an example of the flow of a process performed by the mobile terminal according to the fourth embodiment.
[0027] Fig.16 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to the fifth embodiment.
[0028] Fig.17 is a flowchart showing an example of the flow of a process performed by a mobile terminal according to the fifth embodiment.
[0029] Fig.18 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to the sixth embodiment.
[0030] Fig.19 is a flowchart showing an example of the flow of a process performed by a mobile terminal according to the sixth embodiment.
[0031] Fig. 20 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to a seventh embodiment.
[0032] Fig.21 is a flowchart showing an example of the flow of a process performed by a mobile terminal according to the seventh embodiment.
[0033] Fig. 22 An overview of an information processing apparatus according to an eighth embodiment is shown.
[0034] Fig.23 is a functional block diagram showing an example of a functional configuration of an information processing apparatus according to the eighth embodiment.
[0035] Fig.24 is a flowchart showing an example of the flow of processing performed by the information processing apparatus according to the eighth embodiment.
[0036] Fig.25 An overview of an information processing apparatus according to a ninth embodiment is shown.
[0037] Fig.26 is a functional block diagram showing an example of a functional configuration of an information processing apparatus according to the ninth embodiment.
[0038] Fig. 27 is a flowchart showing an example of the flow of processing performed by the information processing apparatus according to the ninth embodiment. DETAILED DESCRIPTION
[0039] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following embodiments, the same parts are represented by the same reference numerals, and redundant descriptions will be omitted. The present disclosure will be described in the following order of items.
[0040] 1. First Implementation
[0041] 1-1. Prerequisite description - 3D model
[0042] 1-2. Prerequisite Description - AR Application
[0043] 1-3. Overview of mobile terminals
[0044] 1-4. Hardware configuration of mobile terminal
[0045] 1-5. Functional configuration of mobile terminals
[0046] 1-6.Method for measuring line of sight direction
[0047] 1-7. How to change the drawing direction of a 3D model
[0048] 1-8. Flow of processing performed by mobile terminal
[0049] 1-9. Effects of the first embodiment 2. Second embodiment
[0050] 2-1. Overview of Second Embodiment
[0051] 2-2. Flow of processing performed by the mobile terminal
[0052] 2-3. Effects of the Second Embodiment 3. Third Embodiment
[0053] 3-1. Functional configuration of mobile terminals
[0054] 3-2. Flow of processing performed by the mobile terminal
[0055] 3-3. Effects of the third embodiment 4. Fourth embodiment
[0056] 4-1. Functional configuration of mobile terminal
[0057] 4-2. Flow of processing performed by the mobile terminal
[0058] 4-3. Effects of the Fourth Embodiment 5. Fifth Embodiment
[0059] 5-1. Functional configuration of mobile terminals
[0060] 5-2. Flow of processing performed by the mobile terminal
[0061] 5-3. Effects of the Fifth Embodiment
[0062] 6. Sixth Implementation
[0063] 6-1. Functional configuration of mobile terminal
[0064] 6-2. Flow of processing performed by the mobile terminal
[0065] 6-3. Effects of the Sixth Embodiment
[0066] 7. Seventh Implementation Method
[0067] 7-1. Functional configuration of mobile terminals
[0068] 7-2. Flow of processing performed by the mobile terminal
[0069] 7-3. Effects of the Seventh Embodiment
[0070] 8. Eighth Implementation
[0071] 8-1. Overview of information processing devices
[0072] 8-2. Functional configuration of information processing device
[0073] 8-3. Flow of Processing Executed by Information Processing Device
[0074] 8-4. Effects of the Eighth Embodiment
[0075] 9. Ninth Implementation Method
[0076] 9-1. Overview of information processing devices
[0077] 9-2. Functional configuration of information processing device
[0078] 9-3. Flow of Processing Executed by Information Processing Device
[0079] 9-4. Effects of the Ninth Embodiment
[0080] (1. First Embodiment)
[0081] Before describing the embodiments of the present disclosure, prerequisites necessary for realizing the embodiments will be described.
[0082] [1-1. Prerequisite description - 3D model]
[0083] Figure 1 An overview of the process from capturing images to generating 3D models is shown.
[0084] like Figure 1 As shown, generation of the 3D model includes: a process of capturing an image of a subject 90 (object) by using a plurality of image capture devices 1 - 1 to 1 - 3 ; and a process of performing 3D modeling to generate a 3D model 90M having 3D information on the subject 90 .
[0085] Specifically, Figure 1 As shown, a plurality of image capturing devices 1 - 1 to 1 - 3 are placed inwardly around a subject 90 existing in the real world so as to surround the subject 90 . Figure 1 An example is shown in which three image capturing devices are provided and the image capturing devices 1-1 to 1-3 are arranged around the subject 90. Figure 1 In the example of , a person performing a predetermined action is defined as the subject 90 .
[0086] 3D modeling is performed by using a plurality of viewpoint images captured from different viewpoint volumes by the three image capture devices 1-1 to 1-3 in a synchronized manner, thereby generating a 3D model of the subject 90 in units of video frames of the three image capture devices 1-1 to 1-3.
[0087] The 3D model is a model having 3D information about the subject 90. The 3D model 90M has mesh data called a polygon mesh in which geometric information about the subject 90 is represented by connections between vertices, and also has texture information and depth information (distance information) corresponding to each polygon mesh. The information that the 3D model 90M has is not limited to this, and the 3D model 90M may have other information.
[0088] The 3D model 90M is subjected to so-called texture mapping in which a texture indicating the color, pattern, or feel of a mesh is pasted according to the mesh position. In texture mapping, in order to improve the realism of the 3D model 90M, it is desirable to paste a view-dependent texture. However, a view-independent texture may be pasted to prevent an increase in the amount of calculation.
[0089] The generated 3D model 90M is given an identification number corresponding to the attribute (feature) possessed by the 3D model 90M. For example, in the case where the 3D model 90M is a person, attribute items such as the facial direction, the sight direction, and the facial expression of the person are given respective identification numbers indicating the attribute values of the attributes. The given identification numbers are centrally managed in a predetermined attribute table. In the case where the attribute values of the predetermined attribute items are given from the outside, the 3D model 90M corresponding to the given attribute values of the attribute items is read by referring to the attribute table. Note that there is no limitation on the specific format of the attribute table.
[0090] The content data including the read 3D model is transmitted to the reproduction device and reproduced in the reproduction device. The content data including the 3D model is reproduced to present the 3D model, thereby displaying a 3D video on a viewing device of a viewer.
[0091] exist Figure 1 In the example of , a mobile terminal 80 such as a smart phone or a tablet terminal is used as a viewing device. A captured image I is displayed on a display screen 82 of the mobile terminal 80. In the captured image I, a subject 92 captured by a camera 84 included in the mobile terminal 80 is displayed. In the captured image I, in response to the operation of an AR application (described later) running in the mobile terminal 80, a 3D model 90M is displayed while being superimposed. The details will be described later.
[0092] [1-2. Prerequisite description - AR application]
[0093] Figure 2 This is an illustration of an AR application.
[0094] The augmented reality (AR) application is an application that runs in the mobile terminal 80 and displays the 3D model 90M on the display screen 82 of the mobile terminal 80 while superimposing the 3D model 90M when certain triggers are generated.
[0095] Specifically, Figure 2 As shown, in the case where an AR marker 85 serving as a predetermined display is detected from a captured image I monitored by the mobile terminal 80, a 3D model 90M is displayed at a position corresponding to the AR marker 85. That is, the above trigger is the detection of a predetermined display from the captured image I. The AR marker 85 may have any form such as a character string, a symbol string, a graphic, a picture, a barcode, or a QR code, and is not limited to these forms. The AR marker 85 may also be a 2D display or a 3D display.
[0096] In addition, detection of a predetermined target (structure or natural object) obtained by performing spatial analysis or the like on a captured image monitored by the mobile terminal 80 on the display screen 82 can be used as a trigger (without a marker). In addition, whether the current position acquired by the mobile terminal 80 through the GPS receiver is a predetermined position can be used as a trigger.
[0097] [1-3. Overview of mobile terminals]
[0098] Will refer to Figure 3 An overview of a mobile terminal 80 according to this embodiment is described. The mobile terminal 80 is an example of an information processing device in the present disclosure. Figure 3 80 is a first diagram showing an example of an operating state of a mobile terminal according to a first embodiment. The mobile terminal 80 is, for example, a smart phone operated by a user. The mobile terminal 80 includes a display screen 82 for displaying images and videos and a camera 84 arranged near the display screen 82 to face the same direction as the display screen 82.
[0099] The above-mentioned AR application is installed on the mobile terminal 80. When the AR application is executed, the mobile terminal 80 displays the captured image I captured by the camera 18 on the display screen 82 in real time (so-called live view), and determines whether the AR marker 85 appears in the captured image I. In the case where the AR marker 85 appears in the captured image I, the mobile terminal 80 displays the 3D model 90M while superimposing the 3D model 90M at the position corresponding to the AR marker 85.
[0100] In addition, the mobile terminal 80 detects features of the subject 92 appearing in the captured image I, and changes the display mode of the 3D model 90M according to the detected features. Note that a representative scenario using the mobile terminal 80 is assumed to be a scenario in which the user himself serves as the subject 92 and takes a picture of the user himself (self-portrait) by using the camera 84 of the mobile terminal 80.
[0101] In this embodiment, the mobile terminal 80 detects the sight direction E of the subject 92 appearing in the captured image I. When the mobile terminal 80 determines that the sight direction E of the subject 92 faces the camera 84 , the mobile terminal 80 directs the sight direction F of the 3D model 90M toward the camera 84 .
[0102] Then, the mobile terminal 80 takes a self-photograph when both the line-of-sight direction E of the subject 92 and the line-of-sight direction F of the 3D model 90M are directed toward the camera 84 .
[0103] Therefore, the mobile terminal 80 can cause the 3D model 90M to perform a movement or reaction in response to the movement or state of the subject 92. As described above, causing the 3D model 90M to perform some reaction or state transition will be referred to as interactive 3D model AR.
[0104] Figure 4 80 detects the sight line direction E of the subject 92 appearing in the captured image I displayed on the own display screen 82. Then, in the case where the detected sight line direction E faces the camera 84, the mobile terminal 80 selects the 3D model 90N whose sight line direction F faces the camera 84, and displays the 3D model 90N while superimposing the 3D model 90N on the captured image I.
[0105] Thereafter, the mobile terminal 80 takes a selfie when both the subject 92 and the 3D model 90N are looking at the camera. As described above, when the sight line direction E serving as a feature of the subject 92 faces the camera 84, the display mode of the 3D model 90N can be changed so that the sight line direction F of the 3D model 90N faces the camera 84.
[0106] [1-4. Mobile terminal hardware configuration]
[0107] Figure 5 1 is a hardware block diagram showing an example of the hardware configuration of the mobile terminal according to the first embodiment. Figure 4Only the elements related to this embodiment are shown among the hardware components included in the mobile terminal 80 of this embodiment. That is, the mobile terminal 80 is configured so that a central processing unit (CPU) 32, a read-only memory (ROM) 34, a random access memory (RAM) 36, a storage unit 38 and a communication interface 40 are connected via an internal bus 39.
[0108] The CPU 32 controls the entire operation of the mobile terminal 80 by developing the control program P1 stored in the storage unit 38 or the ROM 34 onto the RAM 36 and executing the control program P1. That is, the mobile terminal 80 has a configuration of a general-purpose computer that operates according to the control program P1. The control program P1 can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. The mobile terminal 80 can also perform a series of processes by hardware. The control program P1 executed by the CPU 32 may be a program that performs processes in a time series in the order described in the present disclosure, or may be a program in which processes are performed in parallel or at a necessary timing, such as when called.
[0109] The storage unit 38 includes, for example, a flash memory, and stores the control program P1 executed by the CPU 32 and information such as the 3D model M. Since the 3D model M generally has a large capacity, the 3D model M may be downloaded from an external server (not shown) connected to the mobile terminal 80 via the Internet or the like and stored in the storage unit 38 as needed.
[0110] The communication interface 40 acquires operation information about the touch screen 14 stacked on the liquid crystal display 16 forming the display screen 82 of the mobile terminal 80 via the touch screen interface 42. The communication interface 40 displays image information and video information on the liquid crystal display 16 via the display interface 43. The communication interface 40 controls the operation of the camera 84 via the camera interface 44.
[0111] The communication interface 40 communicates via wireless or wired communication with an external server (not shown) etc. Thus, for example, the mobile terminal 80 receives a newly created 3D model M etc. and transmits a generated captured image I to the outside.
[0112] [1-5. Functional configuration of mobile terminal]
[0113] Figure 6 3 is a functional block diagram showing an example of the functional configuration of the mobile terminal according to the first embodiment. The CPU 32 of the mobile terminal 80 develops the control program P1 onto the RAM 36 and executes the control program P1, thereby realizing Figure 6 4 , there are shown an image capturing unit 51 , an AR marker detecting unit 52 , a subject feature detecting unit 53 , and a display control unit 54 as functional units.
[0114] The image capture unit 51 monitors video, captures images, and records video by using a camera 84 included in the mobile terminal 80. The image capture unit 51 also includes a video monitoring unit 51a and an image capture / video recording processing unit 51b. The camera 84 is an actual camera existing in the real world and is an example of a first image capture unit of the present disclosure.
[0115] The video monitoring unit 51a performs a so-called live view in which the video observed by the camera 84 is displayed in real time on the display screen 82 of the mobile terminal D1.
[0116] The image capturing / video recording processing unit 51b stores the captured image I captured by the camera 84. Furthermore, the image capturing / video recording processing unit 51b performs self-photography.
[0117] The AR marker detection unit 52 detects the AR marker 85 from the captured image I monitored by the process of the video monitoring unit 51a. Specifically, the AR marker detection unit 52 performs template matching or the like to detect a predetermined AR marker 85 determined in advance.
[0118] The subject feature detection unit 53 detects a subject displayed simultaneously with the 3D model 90M in the captured image I monitored by the processing of the video monitoring unit 51a. Then, the subject feature detection unit 53 detects features of the detected subject 92. The subject feature detection unit 53 includes a subject detection unit 53a and a sight line direction detection unit 53b. The subject feature detection unit 53 is an example of a detection unit in the present disclosure.
[0119] The subject detection unit 53 a detects the subject 92 displayed simultaneously with the 3D model 90M in the captured image I monitored by the image capture unit 51 .
[0120] The line-of-sight direction detection unit 53b detects the line-of-sight direction E of the subject 92 detected by the subject detection unit 53a. The line-of-sight direction E is an example of a feature of the subject 92 in the present disclosure.
[0121] The display control unit 54 controls the 3D model 90M displayed in the captured image I. More specifically, the display control unit 54 changes the display mode of the 3D model 90M according to the line of sight direction E of the subject 92 detected by the subject feature detection unit 53. In this embodiment, in a case where the line of sight direction E of the subject 92 faces the camera 84, that is, in a case where the subject 92 is looking at the camera, the display control unit 54 changes the line of sight direction F of the 3D model 90M to face the camera 84. That is, the line of sight direction F is an example of a display mode in the present disclosure. The display control unit 54 includes a 3D model frame selection unit 54a and a 3D model drawing direction control unit 54b. The display control unit 54 is an example of a control unit in the present disclosure.
[0122] The 3D model frame selection unit 54a selects a predetermined default 3D model 90M from the plurality of 3D models M stored in the storage unit 38 based on the AR marker 85. The 3D model frame selection unit 54a also selects a 3D model 90M having a line of sight direction F corresponding to the line of sight direction E of the subject 92 detected by the line of sight direction detection unit 53b from the plurality of 3D models M stored in the storage unit 38. More specifically, in the case where the subject 92 is looking at the camera 84, the 3D model frame selection unit 54a selects a 3D model 90M whose line of sight direction F faces the camera 84.
[0123] The 3D model drawing direction control unit 54b displays the default 3D model 90M selected by the 3D model frame selection unit 54a while superimposing the 3D model 90M on the captured image I displayed on the display screen 82 of the mobile terminal 80. The 3D model drawing direction control unit 54b rotates the 3D model 90M to change the direction of the 3D model 90M. The 3D model drawing direction control unit 54b also rotates the coordinate system in which the 3D model 90M is placed to change the direction of the 3D model 90M, thereby generating a 3D model 90N. The 3D model drawing direction control unit 54b displays the 3D model 90N whose direction has been changed while superimposing the 3D model 90N on the captured image I displayed on the display screen 82 of the mobile terminal 80.
[0124] [1-6. Method for measuring line of sight direction]
[0125] Figure 7 2 is an explanatory diagram of a method of measuring a gaze direction. Various methods are proposed for detecting the gaze direction E from a facial image of the subject 92, and any of these methods can be used in this embodiment.
[0126] Figure 7 Three representative methods of measuring gaze direction are shown. Figure 7A shows a method of capturing an image of the face of the subject 92 by using the camera 84, detecting an eyeball region from the captured face image, and detecting a sight line direction E based on shape features of the detected eyeballs.
[0127] Figure 7 A shows an enlarged view of the left eye of the subject 92 detected from the captured image I captured by the camera 84. The distance dp between the center of the pupil P and the inner corner Q changes according to the line of sight. Specifically, as Figure 7 As shown in A, when the line of sight is directed to the left (towards the outer corner of the eye S), the distance dp increases. At the same time, when the line of sight is directed to the right (towards the inner corner of the eye Q), the distance dp decreases. Therefore, the approximate line of sight direction can be calculated by measuring the distance dp.
[0128] Figure 7 B shows an example in which a facial image of a subject 92 is captured by the camera 84 while the face is illuminated with the infrared LED 86. It is assumed that the camera 84 has sensitivity to light having a wavelength emitted by the infrared LED 86. Figure 7 The method of measuring the line of sight in B is to detect the eyeball area from the captured facial image, and detect the line of sight direction E based on the position of the reflected image (corneal reflected image) of the detected cornea of the eyeball irradiated by the infrared LED 86. Since the wavelength of the light emitted by the infrared LED 86 is invisible, the line of sight direction can be detected without interfering with the subject 92.
[0129] Figure 7 B shows an enlarged view of the left eye of the subject 92 detected from the captured image I captured by the camera 84. The positional relationship between the center of the pupil P and the cornea reflected image R varies depending on the line of sight direction. Specifically, as Figure 7 As shown in FIG. 1B , when the line of sight is directed to the left (toward the outer corner S), the pupil P is located closer to the outer corner S than the cornea reflected image R. At the same time, when the line of sight is directed to the right (toward the inner corner Q), the pupil P is located closer to the inner corner Q than the cornea reflected image R. Therefore, the approximate line of sight direction can be detected by measuring the positional relationship between the pupil P and the cornea reflected image R.
[0130] exist Figure 7 In C, an image of the face of the subject 92 is captured by the camera 84, and the line of sight direction is detected based on the direction of the captured face. In the case where a person quickly or significantly moves the direction of his or her line of sight, the person usually changes the line of sight by moving the head rather than moving his or her eyeballs. Therefore, it can be considered that the direction of the face is roughly consistent with the line of sight in many cases. In the case where the above approximation holds, it can be detected by using Figure 7 C method to detect the gaze direction.
[0131] like Figure 7 As shown in the left part of FIG. C, when the subject 92 faces forward, the distance a1 between the center line C1 and the straight line C2 is substantially equal to the distance a2 between the center line C1 and the straight line C3. The center line C1 passes through the face of the subject 92 in the longitudinal direction, the straight line C2 passes through the right eye and is parallel to the center line C1, and the straight line C3 passes through the left eye and is parallel to the center line C1.
[0132] like Figure 7 As shown in the middle part of C, when the subject 92 faces the right direction, the distance a1 is greater than the distance a2. Figure 7 As shown in the right part of C, when the subject 92 faces the left direction, the distance a1 is smaller than the distance a2.
[0133] As described above, the approximate sight line direction can be detected by comparing the size relationship between the distance a1 and the distance a2.
[0134] Note that there are individual differences in the shape and movement of the eyeball. Therefore, recently, a method using a highly robust eyeball model created by using a method such as machine learning is used to stably detect the eyeball of the subject 92.
[0135] [1-7. How to change the drawing direction of a 3D model]
[0136] Figure 8 A method of changing the drawing direction of the 3D model is shown. Since the 3D model 90M has directionality, when the 3D model 90M is drawn on the display screen 82, the 3D model 90M is drawn in a default direction, for example, the positive direction of the Z axis. In this embodiment, the direction of the 3D model 90M is changed according to the sight line direction E serving as a feature of the subject 92, and the 3D model 90M is drawn as the 3D model 90N. Therefore, in this embodiment, the two methods of controlling the drawing direction described below are used in combination.
[0137] like Figure 8 As shown in A, the first method is a method of rotating the 3D model 90M itself in the target direction to obtain the 3D model 90N.
[0138] like Figure 8 As shown in FIG. 2B , the second method is a method of rotating the coordinate system XYZ in which the 3D model 90M is placed in the target direction and drawing the 3D model 90N on the rotated coordinate system.
[0139] Any method may be used in this embodiment. The method of rotating the coordinate system XYZ where the 3D model 90M is placed requires a small amount of calculation, but in the case where the 3D model 90M has a texture that is independent of the view, the authenticity of the 3D model 90M may be reduced. Meanwhile, the method of rotating the 3D model 90M itself requires a large amount of calculation, but in the case where the 3D model 90M has a texture that is dependent on the view, the authenticity of the 3D model 90M is maintained. Therefore, it is desirable to change the drawing direction with full consideration of which method is applied.
[0140] For example, in the case where the rotation amount of the 3D model 90M is small, it is desirable to apply the method of rotating the coordinate system because the reduction in the authenticity of the 3D model 90M due to the rotation is considered to be small. Meanwhile, in the case where the rotation amount of the 3D model 90M is large, it is desirable to apply the method of rotating the 3D model 90M itself to maintain the authenticity of the 3D model 90M.
[0141] [1-8. Flow of processing performed by mobile terminal]
[0142] Fig. 9 is a flowchart showing an example of the flow of a process performed by the mobile terminal according to the first embodiment. Hereinafter, the flow of the process will be described in sequence.
[0143] The video monitoring unit 51a activates the camera 84 (step S10). Hereinafter, the camera 84 is in a live view of monitoring the captured image I in real time.
[0144] The AR marker detection unit 52 detects the AR marker 85 from the captured image I (step S11 ).
[0145] Then, the AR marker detection unit 52 determines whether the AR marker 85 has been detected (step S12). In the case where it is determined that the AR marker 85 has been detected (step S12: Yes), the process proceeds to step S13. Meanwhile, in the case where it is determined that the AR marker 85 has not been detected (step S12: No), the process returns to step S11.
[0146] In the case where the determination result is “Yes” in step S12, the 3D model frame selection unit 54a selects a predetermined default 3D model 90M from the 3D models M stored in the storage unit 38 based on the AR marker 85. Then, the 3D model drawing direction control unit 54b displays the selected 3D model 90M while superimposing the 3D model 90M at the position corresponding to the AR marker 85 in the captured image I (step S13).
[0147] Next, the subject detection unit 53 a detects the subject 92 from the captured image I (step S14 ).
[0148] The subject detection unit 53a determines whether the subject 92 has been detected (step S15). In the case where it is determined that the subject 92 has been detected (step S15: Yes), the process proceeds to step S16. Meanwhile, in the case where it is determined that the subject 92 has not been detected (step S15: No), the process returns to step S14.
[0149] In a case where the determination result is “Yes” in step S15 , the sight line direction detection unit 53 b detects the sight line direction of the subject 92 (step S16 ).
[0150] Then, the sight line direction detection unit 53b determines whether the subject 92 is looking at the camera 84, that is, whether the sight line direction E of the subject 92 faces the camera 84 (step S17). In the case where it is determined that the sight line direction E of the subject 92 faces the camera 84 (step S17: Yes), the process proceeds to step S19. Meanwhile, in the case where it is determined that the sight line direction E of the subject 92 does not face the camera 84 (step S17: No), the process proceeds to step S18.
[0151] In the case where the result of the determination in step S17 is "No", the 3D model frame selection unit 54a selects the moving image frame of the 3D model 90M from the 3D models M stored in the storage unit 38. Then, the 3D model drawing direction control unit 54b performs loop reproduction of the moving image frame selected by the 3D model frame selection unit 54a (step S18). Thereafter, the process returns to step S16.
[0152] Meanwhile, in the case where the result of the determination in step S17 is "yes", the 3D model drawing direction control unit 54b directs the direction of the 3D model 90M toward the camera 84 (step S19). The 3D model drawing direction control unit 54b may set the direction of the 3D model 90M by rotating the direction of the coordinate system on which the 3D model 90M is drawn, or may set the direction of the 3D model 90M by rotating the 3D model 90M itself.
[0153] Next, the 3D model drawing direction control unit 54b draws the 3D model 90N whose eyes are open and whose line of sight direction F faces the camera 84 selected by the 3D model frame selection unit 54a at the drawing position set in step S19 (step S20).
[0154] Then, the image capturing / video recording processing unit 51 b performs self-photography while both the subject 92 and the 3D model 90N are looking at the camera (step S21 ).
[0155] [1-9. Effects of the First Embodiment]
[0156] As described above, according to the mobile terminal 80 of the first embodiment, the subject feature detection unit 53 (detection unit) detects the sight line direction E (feature) of the subject 92 displayed simultaneously with the 3D model 90M in the captured image I. Then, the display control unit 54 (control unit) changes the sight line direction F (display mode) of the 3D model 90M so that the 3D model 90M faces the camera 84 according to the sight line direction E of the subject 92 detected by the subject feature detection unit 53.
[0157] This makes it possible to cause the 3D model 90M displayed in the captured image I including the subject 92 to perform interactive movement or reaction of looking at the imaging device together with the subject 92 .
[0158] In addition, according to the mobile terminal 80 of the first embodiment, the subject feature detection unit 53 (detection unit) detects the line of sight direction E serving as a feature of the subject 92, and the display control unit 54 (control unit) changes the line of sight direction F (facial direction) of the 3D model 90M according to the detected line of sight direction E of the subject 92.
[0159] This makes it possible to cause the 3D model 90M to perform interactive movement or reaction according to the line of sight direction E of the subject 92 .
[0160] Furthermore, according to the mobile terminal 80 of the first embodiment, the display control unit 54 (control unit) changes the orientation of the 3D model 90M by rotating the 3D model 90M according to the line-of-sight direction E of the subject 92 .
[0161] Therefore, even in the case where the 3D model 90M is rotated, the authenticity of the 3D model 90M having the view-dependent texture can be maintained.
[0162] Furthermore, according to the mobile terminal 80 of the first embodiment, the display control unit 54 (control unit) changes the orientation of the 3D model 90M by rotating the coordinate system in which the 3D model 90M is placed according to the line-of-sight direction E of the subject 92 .
[0163] This makes it possible to perform the process of rotating the 3D model 90M at high speed with a small amount of calculation.
[0164] In addition, according to the mobile terminal 80 of the first embodiment, when the subject feature detection unit 53 (detection unit) detects that the line of sight direction E of the subject 92 is facing the direction in which the captured image I was captured, the display control unit 54 (control unit) changes the direction of the 3D model 90M to the direction in which the captured image I was captured.
[0165] This makes it possible to cause the 3D model 90M to perform interactive movements or reactions in response to the subject 92 .
[0166] Furthermore, in the mobile terminal 80 of the first embodiment, the image capturing unit 51 captures an image of the subject 92 and the 3D model 90M whose display mode has been changed by the display control unit 54 (control unit).
[0167] This makes it possible to take selfies of the subject 92 and the 3D model 90M by using the camera 84, for example.
[0168] Furthermore, in the mobile terminal 80 of the first embodiment, the image capturing unit 51 is a camera 84 existing in the real world or a virtual camera 84V existing in the virtual world (see the eighth embodiment).
[0169] This makes it possible to take a self-photograph of the subject 92 and the 3D model 90M by using the camera 84. Furthermore, it is possible to take a self-photograph of the avatar 92a and the 3D model as a virtual self of the subject 92 by using the virtual camera 84V, which will be described in detail in the eighth embodiment.
[0170] Furthermore, according to the mobile terminal 80 of the first embodiment, the 3D model 90M is a model having 3D information on the subject 90 (object) existing in the real world, the 3D information being generated by using a plurality of viewpoint images of the subject 90 captured from different viewpoints.
[0171] This makes it possible to generate a 3D model 90M that can be observed with the same quality from a plurality of different directions. Therefore, the direction of the 3D model 90M displayed while being superimposed on the captured image I can be freely changed while maintaining the display quality of the 3D model 90M.
[0172] Furthermore, according to the mobile terminal 80 of the first embodiment, when the AR marker 85 (predetermined display) is displayed in the captured image I, the display control unit 54 (control unit) displays the 3D model 90M while superimposing the 3D model 90M on the captured image I.
[0173] This makes it possible to easily and quickly display the 3D model 90M without performing complicated operations or processes.
[0174] (2. Second Embodiment)
[0175] The second embodiment of the present disclosure is an example of a mobile terminal 80 (information processing device) having a function of making a subject 92 and a 3D model 90M look at each other.
[0176] The description of the hardware configuration and functional configuration of the mobile terminal 80 of this embodiment will be omitted because those configurations are the same as those of the mobile terminal 80 described in the first embodiment. However, the control program stored in the storage unit 38 is the same as the control program P1 (see Figure 5 ) are different so as to bring into play the functions unique to this implementation mode.
[0177] [2-1. Overview of Second Embodiment]
[0178] Fig.10 An example of the operating state of the mobile terminal according to the second embodiment is shown. The mobile terminal 80 detects the sight line direction E of the subject 92 appearing in the captured image I displayed on its own display screen 82. Then, in the case where the detected sight line direction E faces the vicinity of the AR marker 85, the mobile terminal 80 selects the 3D model 90N whose sight line direction F faces the subject 92, and displays the 3D model 90N while superimposing the 3D model 90N on the captured image I.
[0179] Then, the mobile terminal 80 takes a selfie while the subject 92 and the 3D model 90N are looking at each other. As described above, when the sight line direction E used as a feature of the subject 92 faces the vicinity of the AR marker 85, the display mode of the 3D model 90N can be changed so that the sight line direction F of the 3D model 90N faces the subject 92.
[0180] [2-2. Flow of processing performed by mobile terminal]
[0181] Fig.11 is a flowchart showing an example of the flow of a process performed by the mobile terminal according to the first embodiment. Hereinafter, the flow of the process will be described in sequence.
[0182] Will omit Fig.11 The description of the process from step S30 to step S36 in the embodiment is omitted because the process is the same as that described in the first embodiment. Fig. 9 The processing flow from step S10 to step S16 is the same.
[0183] After the line-of-sight direction detection unit 53b detects the line-of-sight direction of the subject 92 in step S36, the line-of-sight direction detection unit 53b determines whether the subject 92 faces the AR marker 85 (step S37). In the case where it is determined that the line-of-sight direction of the subject 92 faces the AR marker 85 (step S37: Yes), the process proceeds to step S39. Meanwhile, in the case where it is determined that the line-of-sight direction of the subject 92 does not face the AR marker 85 (step S37: No), the process proceeds to step S38.
[0184] In the case where the result of the determination in step S37 is "No", the 3D model frame selection unit 54a selects the moving image frame of the 3D model 90M from the 3D models M stored in the storage unit 38. Then, the 3D model drawing direction control unit 54b performs loop reproduction of the moving image frame selected by the 3D model frame selection unit 54a (step S38). Thereafter, the process returns to step S36.
[0185] Meanwhile, in the case where the determination result is “Yes” in step S37 , the 3D model drawing direction control unit 54 b directs the drawing position of the 3D model 90M toward the face of the subject 92 (step S39 ).
[0186] Next, the 3D model drawing direction control unit 54b draws the 3D model 90N of the face whose eyes are open and whose line of sight faces the subject 92 selected by the 3D model frame selection unit 54a at the drawing position set in step S39 (step S40).
[0187] Then, the image capturing / video recording processing unit 51 b performs self-photography while the subject 92 and the 3D model 90N are looking at each other (step S41 ).
[0188] [2-3. Effects of the Second Embodiment]
[0189] As described above, according to the mobile terminal 80 of the second embodiment, when the subject feature detection unit 53 (detection unit) detects that the line of sight direction E of the subject 92 is facing the 3D model 90M, the display control unit 54 (control unit) draws the 3D model 90N whose direction has changed to facing the subject 92.
[0190] Therefore, it is possible to take a selfie while the subject 92 and the 3D model 90N are looking at each other.
[0191] (3. Third Embodiment)
[0192] The third embodiment of the present disclosure is an example of a mobile terminal 80 (information processing device) having a function of taking a selfie while making a 3D model 90M face the camera 84 with a smile, in a case where a subject 92 is looking at the camera 84 with a smile. That is, the mobile terminal 80 of this embodiment changes the display mode (the direction of sight and the facial expression) of the 3D model 90M according to the direction of sight and the facial expression serving as the characteristics of the subject 92.
[0193] The description of the hardware configuration of the mobile terminal 80 of this embodiment will be omitted because the configuration is the same as that of the mobile terminal 80 described in the first embodiment. Note that the control program stored in the storage unit 38 is the same as the control program P1 (see Figure 5 ) are different so as to bring into play the functions unique to this implementation mode.
[0194] [3-1. Functional configuration of mobile terminal]
[0195] Fig.12 1 is a functional block diagram showing an example of the functional configuration of the mobile terminal according to the third embodiment. The mobile terminal 80 of the third embodiment has a function of providing a mobile terminal 80 (see FIG. 1 ) with a first embodiment. Figure 6 ) A functional configuration obtained by adding some functions. Hereinafter, only the difference in functional configuration from the mobile terminal 80 of the first embodiment will be described.
[0196] That is, the subject feature detection unit 53 includes a smile detection unit 53c in addition to the subject detection unit 53a and the sight line direction detection unit 53b.
[0197] The smile detection unit 53c detects whether the subject 92 appearing in the captured image I is smiling. The smile detection unit 53c extracts the facial area of the subject 92 detected by the subject detection unit 53a. Then, the smile detection unit 53c compares the extracted facial area with the facial expression database ( Fig.12 In the case where the extracted facial region indicates a high degree of verification with the sample indicating a smile, the smile detection unit 53c determines that the subject 92 is smiling.
[0198] Note that the smile detection unit 53c may determine whether the subject is smiling by using other image processing algorithms.
[0199] [3-2. Flow of processing performed by mobile terminal]
[0200] Fig.13 : is a flowchart showing an example of the flow of a process performed by the mobile terminal according to the third embodiment. Hereinafter, the flow of the process will be described in sequence.
[0201] Will omit Fig.13 The description of the process from step S50 to step S55 in the embodiment is omitted because the process is the same as that described in the first embodiment. Fig. 9 The processing flow from step S10 to step S15 is the same.
[0202] In the case where it is determined in step S55 that the subject 92 has been detected (step S55: Yes), the line of sight direction detection unit 53b detects the line of sight direction E of the subject 92. Then, the smile detection unit 53c determines whether the subject 92 smiles (step S56).
[0203] Then, in the case where it is determined in step S56 that the subject 92 is smiling, the sight line direction detection unit 53b determines whether the sight line direction E of the subject 92 is facing the camera 84 (step S57). In the case where it is determined that the subject 92 is smiling and the sight line direction E is facing the camera 84 (step S57: Yes), the processing proceeds to step S59. Meanwhile, in the case where it is determined that the subject 92 is not smiling and the sight line direction E is not facing the camera 84 (step S57: No), the processing proceeds to step S58.
[0204] In the case where the result of the determination in step S57 is "No", the 3D model frame selection unit 54a selects the moving image frame of the 3D model 90M from the 3D models M stored in the storage unit 38. Then, the 3D model drawing direction control unit 54b performs loop reproduction of the moving image frame selected by the 3D model frame selection unit 54a (step S58). Thereafter, the process returns to step S56.
[0205] Meanwhile, in the case where the determination result in step S57 is “Yes”, the 3D model drawing direction control unit 54 b directs the drawing position of the 3D model 90M toward the camera 84 (step S59 ).
[0206] Next, the 3D model drawing direction control unit 54b draws the 3D model 90N with its eyes open and its line of sight F facing the camera 84 and smiling, selected by the 3D model frame selection unit 54a, at the drawing position set in step S59 (step S60).
[0207] Then, the image capturing / video recording processing unit 51 b performs self-photography while both the subject 92 and the 3D model 90N are looking at the camera (step S61 ).
[0208] [3-3. Effects of the Third Embodiment]
[0209] As described above, according to the mobile terminal 80 of the third embodiment, the subject feature detection unit 53 (detection unit) detects a smile (facial expression) serving as a feature of the subject 92. Then, in the case where it is determined that the facial expression of the subject 92 is a smile, the display control unit 54 (control unit) changes the direction of the 3D model 90M to the direction toward the camera 84, that is, the direction in which the captured image I is captured.
[0210] Therefore, it is possible to take a selfie while both the subject 92 and the 3D model 90N are looking at the camera with a smile.
[0211] (4. Fourth Embodiment)
[0212] The fourth embodiment of the present disclosure is an example of a mobile terminal 80 (information processing device) having a function of taking a selfie after automatically correcting the brightness and skin color of a 3D model 90M according to the image capturing environment of a subject 92. That is, the mobile terminal 80 of this embodiment changes the display mode (brightness) of the 3D model 90M according to the brightness (e.g., brightness of the skin) serving as a feature of the subject 92.
[0213] The description of the hardware configuration of the mobile terminal 80 of this embodiment will be omitted because the configuration is the same as that of the mobile terminal 80 described in the first embodiment. Note that the control program stored in the storage unit 38 is the same as the control program P1 (see Figure 5 ) are different so as to bring into play the functions unique to this implementation mode.
[0214] [4-1. Functional configuration of mobile terminal]
[0215] Fig.14 1 is a functional block diagram showing an example of the functional configuration of a mobile terminal according to a fourth embodiment. The mobile terminal 80 of the fourth embodiment has a configuration obtained by combining the mobile terminal 80 of the first embodiment (see Figure 6 ) is replaced with some functions. Hereinafter, only the difference in functional configuration from the mobile terminal 80 of the first embodiment will be described.
[0216] The subject feature detection unit 53 includes an image capturing environment evaluation unit 53d instead of the line of sight direction detection unit 53b.
[0217] The image capturing environment evaluation unit 53d measures the brightness of the image capturing environment of the captured image I and the brightness of the subject 92 appearing in the captured image I. For example, the image capturing environment evaluation unit 53d divides the captured image I into regions, that is, the region of the subject 92 and the region other than the subject 92, and obtains the average brightness of each region. The image capturing environment evaluation unit 53d can measure the skin color of the subject 92.
[0218] The display control unit 54 includes a 3D model brightness control unit 54 c instead of the 3D model drawing direction control unit 54 b .
[0219] The 3D model brightness control unit 54c corrects the brightness of the texture of the 3D model 90M. For example, the 3D model brightness control unit 54c compares the brightness of the image capturing environment measured by the image capturing environment evaluation unit 53d with the brightness of the environment in which the 3D model 90M is generated. Then, in a case where the brightness of the environment in which the 3D model 90M is generated is brighter than the brightness of the image capturing environment measured by the image capturing environment evaluation unit 53d, the texture of the 3D model 90M is corrected to be darker. At the same time, in a case where the brightness of the environment in which the 3D model 90M is generated is darker than the brightness of the image capturing environment measured by the image capturing environment evaluation unit 53d, the texture of the 3D model 90M is corrected to be brighter.
[0220] The 3D model brightness control unit 54c can correct the texture of the 3D model 90M based on the skin brightness and skin color of the subject 92 appearing in the captured image I so that the skin brightness and skin color of the 3D model 90M are at the same level as the skin brightness and skin color of the subject 92.
[0221] [4-2. Flow of processing performed by mobile terminal]
[0222] Fig.15 : is a flowchart showing an example of the flow of a process performed by the mobile terminal according to the fourth embodiment. Hereinafter, the flow of the process will be described in sequence.
[0223] Will omit Fig.15 The description of the process from step S70 to step S73 in the embodiment is omitted because the process is the same as that described in the first embodiment. Fig. 9 The processing flow from step S10 to step S13 is the same.
[0224] After step S73, the subject detection unit 53a detects the subject 92 from the captured image I. Then, the image capturing environment evaluation unit 53d detects the brightness of the image capturing environment of the captured image I and the brightness 92 of the subject appearing in the captured image I (step S74).
[0225] Next, the image capturing environment evaluation unit 53d determines whether the subject 92 appears in the captured image I and whether the brightness of the image capturing environment is brighter than the brightness of the image capturing environment of the 3D model 90M (step S75). In the case where it is determined that the above conditions are satisfied (step S75: Yes), the process proceeds to step S76. Meanwhile, in the case where it is determined that the conditions are not satisfied (step S75: No), the process proceeds to step S77.
[0226] In the case where the determination result is "YES" in step S75, the 3D model brightness control unit 54c corrects the texture of the 3D model 90M to a brighter texture (step S76). Thereafter, the process proceeds to step S78.
[0227] Meanwhile, in the case where the determination result in step S75 is “NO”, the 3D model brightness control unit 54 c corrects the texture of the 3D model 90M to a darker texture (step S77 ).
[0228] After step S76 or step S77, the 3D model drawing direction control unit 54b draws the 3D model 90N obtained by performing texture brightness correction corresponding to step S76 or step S77 on the 3D model 90M selected by the 3D model frame selection unit 54a in the captured image I (step S78).
[0229] Then, the image capturing / video recording processing unit 51 b performs self-photography when both the subject 92 and the 3D model 90N appear (step S79 ).
[0230] [4-3. Effects of the Fourth Embodiment]
[0231] As described above, according to the mobile terminal 80 of the fourth embodiment, the subject feature detection unit 53 (detection unit) detects brightness serving as a feature of the subject 92. Then, the display control unit 54 (control unit) changes the brightness 90M of the 3D model according to the brightness of the subject 92.
[0232] Therefore, it is possible to take a selfie with the 3D model 90N whose texture brightness has been corrected according to the facial tone of the subject 92 .
[0233] (5. Fifth Embodiment)
[0234] The fifth embodiment of the present disclosure is an example of a mobile terminal 80 (information processing device) having a function of correcting the drawing position (distance from the camera 84) of the 3D model 90M according to the position (distance from the camera 84) of the subject 92 and taking a selfie when both the subject 92 and the 3D model 90M are in focus. That is, the mobile terminal 80 of this embodiment changes the display mode (drawing position) of the 3D model 90M according to the position (distance from the camera 84) serving as a feature of the subject 92.
[0235] The description of the hardware configuration of the mobile terminal 80 of this embodiment will be omitted because the configuration is the same as that of the mobile terminal 80 described in the first embodiment. Note that the control program stored in the storage unit 38 is the same as the control program P1 (see Figure 5 ) are different so as to bring into play the functions unique to this implementation mode.
[0236] [5-1. Functional configuration of mobile terminal]
[0237] Fig.161 is a functional block diagram showing an example of the functional configuration of a mobile terminal according to the fifth embodiment. The mobile terminal 80 of the fifth embodiment has a configuration obtained by combining the mobile terminal 80 of the first embodiment (see Figure 6 ) is replaced with some functions. Hereinafter, only the difference in functional configuration from the mobile terminal 80 of the first embodiment will be described.
[0238] The subject feature detection unit 53 includes a depth of field calculation unit 53e instead of the line of sight direction detection unit 53b.
[0239] The depth calculation unit 53e calculates the distance from the camera 84 to the subject 92 and the AR marker 85 appearing in the captured image I. The depth calculation unit 53e also calculates the depth of field of the captured image I. The depth calculation unit 53e calculates the depth of field based on the aperture (F value) of the camera 84, the focal length of the lens, and the distance to the subject 92.
[0240] The display control unit 54 includes a 3D model drawing distance control unit 54 d instead of the 3D model drawing direction control unit 54 b .
[0241] The 3D model drawing distance control unit 54d controls the drawing position of the 3D model 90M (the distance from the camera 84 to the position where the 3D model 90M is drawn). More specifically, the 3D model drawing distance control unit 54d controls the drawing position of the 3D model 90M so that the 3D model 90M and the subject 92 are located at the same depth of field.
[0242] [5-2. Flow of processing performed by mobile terminal]
[0243] Fig.17 : is a flowchart showing an example of the flow of a process performed by the mobile terminal according to the fifth embodiment. Hereinafter, the flow of the process will be described in sequence.
[0244] Will omit Fig.17 The description of the process from step S80 to step S83 in the embodiment is omitted because the process is the same as that described in the first embodiment. Fig. 9 The processing flow from step S10 to step S13 is the same.
[0245] After step S83 , the depth calculation unit 53 e calculates the distance from the camera 84 to the AR marker 85 appearing in the captured image I (step S84 ).
[0246] Next, the subject detection unit 53 a detects the subject 92 from the captured image I (step S85 ). At this time, the depth of field calculation unit 53 e calculates the distance from the imaging device 84 to the subject 92 .
[0247] The depth of field calculation unit 53e determines whether the subject 92 appears in the captured image I and whether the AR marker 85 and the subject 92 are located at the same depth of field (step S86). In the case where it is determined that the above conditions are satisfied (step S86: Yes), the process proceeds to step S88. Meanwhile, in the case where it is determined that the conditions are not satisfied (step S86: No), the process proceeds to step S87. Note that whether the AR marker 85 and the subject 92 are located at the same depth of field can be determined based on whether the distance from the AR marker 85 to the subject 92 calculated based on the results of steps S84 and S85 is within the depth of field calculated according to the image capturing conditions of the subject 92.
[0248] When the result determined in step S86 is "No", that is, when the AR marker 85 and the subject 92 are not located at the same depth of field, the 3D model drawing distance control unit 54d draws the drawing position of the 3D model 90M in the captured image I within the distance at which the 3D model 90M and the subject 92 are located at the same depth of field (step S87).
[0249] Next, the 3D model drawing direction control unit 54b draws the 3D model 90M selected by the 3D model frame selection unit 54a at the drawing position set in step S87 (step S88).
[0250] Then, the image capturing / video recording processing unit 51 b performs self-photography when the subject 92 and the 3D model 90M are located at the same depth of field (step S89 ).
[0251] [5-3. Effects of the Fifth Embodiment]
[0252] As described above, according to the mobile terminal 80 of the fifth embodiment, the subject feature detection unit 53 (detection unit) detects the distance from the camera 84 to the subject 92 as a feature of the subject 92. Then, the display control unit 54 (control unit) changes the distance to the 3D model 90M so that the 3D model 90M and the subject 92 are located within the depth of field of the captured image I.
[0253] Therefore, it is possible to take a self-portrait while both the subject 92 and the 3D model 90M are in focus.
[0254] (6. Sixth Implementation Method)
[0255] The sixth embodiment of the present disclosure is an example of a mobile terminal 80 (information processing device) having a function of correcting the drawing position of a 3D model 90M according to the position of a subject 92 in a captured image I and taking a selfie without overlapping the subject 92 and the 3D model 90M. That is, the mobile terminal 80 of this embodiment changes the display mode (drawing position) of the 3D model 90M according to the position serving as a feature of the subject 92.
[0256] The description of the hardware configuration of the mobile terminal 80 of this embodiment will be omitted because the configuration is the same as that of the mobile terminal 80 described in the first embodiment. Note that the control program stored in the storage unit 38 is the same as the control program P1 (see Figure 5 ) are different so as to bring into play the functions unique to this implementation mode.
[0257] [6-1. Functional configuration of mobile terminal]
[0258] Fig.18 1 is a functional block diagram showing an example of the functional configuration of a mobile terminal according to the sixth embodiment. The mobile terminal 80 of the sixth embodiment has a configuration obtained by combining the mobile terminal 80 of the first embodiment (see Figure 6 ) is replaced with some functions. Hereinafter, only the difference in functional configuration from the mobile terminal 80 of the first embodiment will be described.
[0259] The subject feature detection unit 53 includes an overlap determination unit 53f instead of the line of sight direction detection unit 53b.
[0260] The overlap determination unit 53f determines whether the subject 92 overlaps with the AR marker 85 in the captured image I. Specifically, the overlap determination unit 53f detects the position of the subject 92 in the captured image I. Then, the overlap determination unit 53f compares the detected position of the subject 92 with the position of the AR marker 85 detected by the AR marker detection unit 52. In the case where the subject 92 overlaps with the AR marker 85, the overlap determination unit 53f predicts that when the 3D model 90M is drawn at the position corresponding to the AR marker 85 in the captured image I, the subject 92 and the 3D model 90M will overlap.
[0261] The display control unit 54 includes a 3D model drawing position control unit 54 e instead of the 3D model drawing direction control unit 54 b .
[0262] In a case where the overlap determination unit 53 f predicts that the subject 92 and the 3D model 90M will overlap, the 3D model drawing position control unit 54 e draws the 3D model 90M at a position that does not overlap with the position of the subject 92 .
[0263] [6-2. Flow of processing performed by mobile terminal]
[0264] Fig.19 : is a flowchart showing an example of the flow of a process performed by a mobile terminal according to the sixth embodiment. Hereinafter, the flow of the process will be described in sequence.
[0265] Will omit Fig.19The description of the process from step S90 to step S92 in the embodiment is omitted because the process is the same as that described in the first embodiment. Fig. 9 The processing flow from step S10 to step S12 is the same.
[0266] In the case where it is determined in step S92 that the AR marker 85 has been detected (step S92 : Yes), the subject detection unit 53 a detects the subject 92 from the captured image I (step S93 ).
[0267] Next, the overlap determination unit 53 f detects the positional relationship between the AR marker 85 and the subject 92 (step S94 ).
[0268] Furthermore, the overlap determination unit 53f determines whether the subject 92 appears in the captured image I and whether the AR marker 85 does not overlap with the subject 92 (step S95). In the case where it is determined that the above conditions are satisfied (step S95: Yes), the process proceeds to step S97. Meanwhile, in the case where it is determined that the conditions are not satisfied (step S95: No), the overlap determination unit 53f predicts that the subject 92 will overlap with the 3D model 90M when the 3D model 90M is drawn at the position corresponding to the AR marker 85, and performs the process in step S96.
[0269] If the result of determination in step S95 is "No", the 3D model drawing position control unit 54e changes the drawing position of the 3D model 90M that should be drawn at the position corresponding to the AR marker 85 to a position that does not overlap with the position of the subject 92 (for example, a position away from the subject 92) (step S96).
[0270] Then, the 3D model drawing position control unit 54e draws the selected 3D model 90M at the position changed in step S96. Meanwhile, in the case where the determination result in step S95 is "Yes", the 3D model drawing position control unit 54e draws the selected 3D model 90M at the position corresponding to the AR marker 85 (step S97).
[0271] Then, the image capturing / video recording processing unit 51 b performs self-photography in a state where the subject 92 and the 3D model 90M do not overlap (step S98 ).
[0272] [6-3. Effects of the Sixth Embodiment]
[0273] As described above, according to the mobile terminal 80 of the sixth embodiment, the subject feature detection unit 53 (detection unit) detects the position of the subject 92 in the captured image I, which serves as a feature of the subject 92. Then, in a case where the position of the subject 92 and the position of the 3D model 90M are predicted to overlap, the display control unit 54 (control unit) changes the display position of the 3D model 90M to a position that does not overlap with the position of the subject 92.
[0274] This makes it possible to take a selfie when the subject 92 and the 3D model 90M do not overlap.
[0275] (7. Seventh Implementation Method)
[0276] The seventh embodiment of the present disclosure is an example of a mobile terminal 80 (information processing device) having a function of correcting the size and scale of a 3D model 90M according to the size of a subject 92 in a captured image I and taking a selfie so that the subject 92 and the 3D model 90M appear to be the same height, or the face of the subject 92 appears to be smaller than the face of the 3D model 90M. That is, the mobile terminal 80 of this embodiment changes the display mode (size and scale) of the 3D model 90M according to the size serving as a feature of the subject 92.
[0277] The description of the hardware configuration of the mobile terminal 80 of this embodiment will be omitted because the configuration is the same as that of the mobile terminal 80 described in the first embodiment. Note that the control program stored in the storage unit 38 is the same as the control program P1 (see Figure 5 ) are different so as to bring into play the functions unique to this implementation mode.
[0278] [7-1. Functional configuration of mobile terminal]
[0279] Fig. 20 1 is a functional block diagram showing an example of the functional configuration of the mobile terminal according to the sixth embodiment. The mobile terminal 80 of the seventh embodiment has a configuration obtained by combining the mobile terminal 80 of the first embodiment (see Figure 6 ) is replaced with some functions. Hereinafter, only the difference in functional configuration from the mobile terminal 80 of the first embodiment will be described.
[0280] The subject feature detection unit 53 includes a positional relationship detection unit 53g instead of the line of sight direction detection unit 53b.
[0281] The positional relationship detection unit 53g detects the positional relationship between the position of the AR marker 85 detected by the AR marker detection unit 52 and the position of the subject 92 detected by the subject detection unit 53a. The positional relationship detection unit 53g also detects the face size of the subject 92 in consideration of the angle of view of the camera 84.
[0282] The display control unit 54 includes a 3D model drawing position control unit 54 e instead of the 3D model drawing direction control unit 54 b .
[0283] When drawing the 3D model 90M in the captured image I, the 3D model drawing position control unit 54e sets the scale and drawing position of the 3D model 90M. Specifically, the drawing position of the 3D model 90M is set to match the height of the subject 92. The drawing position of the 3D model 90M is also set so that the face position of the subject 92 is at the same height as the face position of the 3D model 90M. Of course, the drawing position may also be set based on conditions other than the above conditions.
[0284] [7-2. Flow of processing performed by mobile terminal]
[0285] Fig.21 : is a flowchart showing an example of the flow of a process performed by the mobile terminal according to the seventh embodiment. Hereinafter, the flow of the process is described in sequence.
[0286] Will omit Fig.21 The description of the process from step S100 to step S102 in the embodiment is omitted because the process is the same as that described in the first embodiment. Fig. 9 The processing flow from step S10 to step S12 is the same.
[0287] In the case where it is determined in step S102 that the AR marker 85 has been detected (step S102 : Yes), the subject detection unit 53 a detects the subject 92 from the captured image I (step S103 ).
[0288] Next, the positional relationship detection unit 53 g detects the positional relationship between the AR marker 85 and the subject 92 (step S104 ).
[0289] The positional relationship detection unit 53g also detects the size of the face of the subject 92 in consideration of the angle of view of the imaging device 84 (step S105).
[0290] Then, the positional relationship detection unit 53g determines whether the captured image I is a portrait picture in which the upper body of the subject 92 appears (step S106). In the case where it is determined that the captured image is a portrait picture (step S106: Yes), the process proceeds to step S107. Meanwhile, in the case where it is determined that the captured image is not a portrait picture (step S106: No), the process proceeds to step S108.
[0291] In the case where the result of the determination in step S106 is "yes", the 3D model drawing position control unit 54e sets the scale and drawing position for drawing the 3D model 90M so that the face position of the 3D model moving image is at the same height as the face position of the subject 92 (step S107). Thereafter, the process proceeds to step S109.
[0292] Meanwhile, in the case where the determination result is “NO” in step S106 , the 3D model drawing position control unit 54 e sets the scale and drawing position of the 3D model moving image to match the height of the subject 92 (step S108 ).
[0293] Then, the 3D model drawing position control unit 54 e draws the selected 3D model 90M in the captured image I under the conditions set in step S107 or step S108 (step S109 ).
[0294] Then, the image capturing / video recording processing unit 51 b performs self-photography while arranging the subject 92 and the 3D model 90M in a well-balanced manner (step S110 ).
[0295] [7-3. Effects of the Seventh Embodiment]
[0296] As described above, according to the mobile terminal 80 of the seventh embodiment, the subject feature detection unit 53 (detection unit) detects the position and size of the subject 92 in the captured image I, which are used as features of the subject 92. Then, the display control unit 54 (control unit) adjusts the display position and size of the 3D model 90M according to the position and size of the subject 92.
[0297] Therefore, it is possible to take a selfie while arranging the subject 92 and the 3D model 90M in a well-balanced manner.
[0298] Note that each of the above embodiments may have the functions of multiple different embodiments. In this case, the mobile terminal 80 includes all functional configurations of the multiple embodiments.
[0299] (8. Eighth Implementation Method)
[0300] The eighth embodiment of the present disclosure is an example of an information processing device 81a having a function of forming a virtual space in a captured image captured by the camera 84 and displaying an avatar and a 3D model 90M of a subject 92 in the virtual space. The avatar of the subject 92 displayed in the captured image can move in the virtual space in response to an operation from the subject 92.
[0301] [8-1. Overview of Information Processing Device]
[0302] Fig. 22An overview of an information processing apparatus according to an eighth embodiment is shown.
[0303] The camera device 84 connected to the information processing device 81a captures an image of a user as a subject 92 and generates a captured image Ia (virtual image). The generated captured image Ia is wirelessly transmitted and displayed on a head-mounted display 89 (HMD) worn by the subject 92. An avatar 92a (first avatar) as a virtual self of the subject 92 is displayed in the captured image Ia displayed on the head-mounted display 89, and the 3D model 90M described in each embodiment of the present disclosure is also displayed while being superimposed on the captured image Ia. The avatar 92a is a complete CG moving image. The head-mounted display 89 has not only a function of detecting the head movement and line of sight direction of the subject 92, but also a sound output function of sending the sound information output by the information processing device 81a to the subject 92.
[0304] The subject 92 also wears a controller 88, which has the function of an acceleration sensor that detects the position, posture, movement, etc. of the subject 92 itself and the function of an operation switch that issues an operation instruction to the information processing device. The line of sight direction and movement of the head of the subject 92 detected by the head-mounted display 89 and the operation information and the movement of the subject 92 detected by the controller 88 are wirelessly transmitted to the information processing device 81a, and are reflected in the movement of the avatar 92a and the operation of the information processing device 81a. That is, the user who is the subject 92 can move freely in the virtual space formed in the captured image Ia while viewing the captured image Ia. That is, the movement of the subject 92 is reflected in the movement of the avatar 92a.
[0305] In particular, in the information processing device 81a of this embodiment, the avatar 92a has a function of capturing an image by using a virtual camera 84V in a virtual space. That is, when the subject 92 issues an instruction to operate the virtual camera 84V via the controller 88, the avatar 92a takes out the virtual camera 84V and captures an image in a direction in the virtual space as instructed by the subject 92.
[0306] When the sight line direction E of the avatar 92a appearing in the captured image 1a captured by the virtual camera 84V faces the virtual camera 84V, the 3D model 90M is changed to a 3D model 90N whose sight line direction F faces the virtual camera 84V. Then, the information processing device 81a takes a selfie while both the avatar 92a and the 3D model 90N are looking at the virtual camera 84V.
[0307] The hardware configuration of the information processing device 81a is obtained by combining the hardware configuration of the mobile terminal 80 described in the first embodiment ( Figure 5) is obtained by replacing the liquid crystal display 16 with a head mounted display 89 and replacing the touch screen 14 with a controller 88 (not shown). The CPU 32 of the information processing device 81a is connected to the head mounted display 89 and the controller 88 by wireless or wired communication.
[0308] [8-2. Functional configuration of information processing device]
[0309] Fig.23 81a of the eighth embodiment includes an image capturing unit 51, a subject feature detecting unit 53, and a display control unit 54. In addition, the information processing device 81a performs a process of generating an avatar 92a of a subject 92 of a complete CG. However, since the process of generating the avatar 92a can be performed according to a known method, its description will be omitted.
[0310] In response to an operation instruction from the subject 92, the image capture unit 51 monitors video, captures images, and records video by using the virtual camera 84V operated by the avatar 92a of the subject 92. The image capture unit 51 includes a virtual video generation unit 51c, a virtual video monitoring unit 51d, a virtual camera video generation unit 51e, and an image capture / video recording processing unit 51b.
[0311] The virtual video generation unit 51c generates a captured image Ia (virtual image) captured by the virtual camera 84V.
[0312] The virtual video monitoring unit 51d performs a so-called live view of displaying the captured image Ia on the head mounted display 89 in real time.
[0313] The image capturing / video recording processing unit 51b stores the captured image Ia captured by the virtual camera 84V. In addition, the image capturing / video recording processing unit 51b performs self-photography.
[0314] The subject feature detection unit 53 detects the avatar 92a of the subject 92 from the captured image Ia captured by the virtual camera 84V. The subject feature detection unit 53 includes an avatar detection unit 53h and a sight line direction detection unit 53b.
[0315] The avatar detection unit 53h detects the avatar 92a from the captured image Ia captured by the virtual camera 84V.
[0316] The sight line direction detection unit 53b detects the sight line direction E of the avatar 92a detected by the avatar detection unit 53h.
[0317] The display control unit 54 changes the display mode of the 3D model 90M according to the sight line direction E of the avatar 92a detected by the subject feature detection unit 53. Specifically, when the sight line direction E of the avatar 92a faces the virtual camera 84V, that is, when the avatar 92a is looking at the virtual camera, the display control unit 54 changes the sight line direction F of the 3D model 90M to face the virtual camera 84V. The display control unit 54 includes a 3D model frame selection unit 54a and a 3D model drawing direction control unit 54b. The display control unit 54 is an example of a control unit in the present disclosure.
[0318] The 3D model frame selection unit 54a selects a predetermined 3D model 90M from the plurality of 3D models M stored in the storage unit 38. The 3D model frame selection unit 54a also selects a 3D model 90M having a line of sight direction F corresponding to the line of sight direction E of the avatar 92a detected by the line of sight direction detection unit 53b. More specifically, in the case where the avatar 92a is looking at the virtual camera, the 3D model frame selection unit 54a selects a 3D model 90M whose line of sight direction F faces the virtual camera 84V.
[0319] The 3D model drawing direction control unit 54b displays the 3D model 90M selected by the 3D model frame selection unit 54a while superimposing the 3D model 90M on the captured image 1a. The 3D model drawing direction control unit 54b rotates the 3D model 90M to change the direction of the 3D model 90M, thereby generating the 3D model 90N. The 3D model drawing direction control unit 54b displays the 3D model 90N whose direction has been changed while superimposing the 3D model 90N on the captured image 1a.
[0320] [8-3. Flow of Processing Executed by Information Processing Device]
[0321] Fig.24 : is a flowchart showing an example of the flow of processing performed by the information processing device according to the eighth embodiment. Hereinafter, the flow of processing will be described in sequence.
[0322] In response to an operation instruction from the controller 88 made by the subject 92, the virtual video monitoring unit 51d activates the virtual camera 84V (step S120).
[0323] The virtual video generation unit 51c generates a captured image Ia (virtual image) viewed from the virtual camera 84V (step S121).
[0324] The avatar detection unit 53h detects the avatar 92a from the captured image Ia. Then, the sight line direction detection unit 53b detects the sight line direction E of the detected avatar 92a (step S122).
[0325] Then, the sight line direction detection unit 53b determines whether the avatar 92a is looking at the virtual camera 84V (step S123). In the case where it is determined that the avatar 92a is looking at the virtual camera (step S123: Yes), the process proceeds to step S125. Meanwhile, in the case where it is determined that the avatar 92a is not looking at the virtual camera (step S123: No), the process proceeds to step S124.
[0326] In the case where the result of the determination in step S123 is "No", the 3D model frame selection unit 54a selects a moving image frame of the 3D model 90M from the 3D models M stored in the storage unit 38. Then, the 3D model drawing direction control unit 54b draws the 3D model 90M selected by the 3D model frame selection unit 54a at a predetermined position in the captured image Ia, and performs loop reproduction of the moving image frame (step S124). Thereafter, the process returns to step S122.
[0327] Meanwhile, if the result of the determination in step S123 is “Yes”, the 3D model drawing direction control unit 54b draws the 3D model 90M at a predetermined position in the captured image Ia, and directs the 3D model 90M toward the virtual camera 84V (step S125).
[0328] Next, the 3D model drawing direction control unit 54b draws the 3D model 90N whose eyes are open and whose line of sight faces the virtual camera 84V, selected by the 3D model frame selection unit 54a, at the drawing position set in step S125 (step S126). Fig. 22 The captured image Ib (virtual image) is shown.
[0329] Then, the image capturing / video recording processing unit 51b takes a self-photograph while both the avatar 92a and the 3D model 90N are looking at the virtual camera (step S127).
[0330] [8-4. Effects of the Eighth Embodiment]
[0331] As described above, according to the information processing device 81a of the eighth embodiment, the subject feature detection unit 53 (detection unit) detects the sight line direction E (feature) of the avatar 92a (first avatar). The avatar 92a exists in the same virtual space as the 3D model 90M displayed in the captured image 1a captured by the virtual camera 84V (second image capture unit) existing in the virtual space, and moves in the virtual space in response to the operation from the subject 92 (user). Then, the display control unit 54 (control unit) changes the display mode of the 3D model 90M according to the detected sight line direction E of the avatar 92a.
[0332] Therefore, the 3D model 90M displayed in the captured image 1a including the avatar 92a (first avatar) of the virtual self as the subject 92 can be caused to perform interactive actions or reactions.
[0333] (9. Ninth Implementation Method)
[0334] The ninth embodiment of the present disclosure is a function of an information processing device having a function of displaying an avatar (second avatar) of a subject different from subject 92 while superimposing the avatar on captured image Ia (virtual image), rather than an example of a function of displaying 3D model 90M while superimposing 3D model 90M on captured image Ia (virtual image) in the eighth embodiment.
[0335] [9-1. Overview of Information Processing Device]
[0336] Fig.25 An overview of an information processing apparatus according to a ninth embodiment is shown.
[0337] The information processing device 81b includes not only the camera 84a having the function described in the eighth embodiment, but also the camera 84b having the same function as the camera 84a. The camera 84b captures an image of a subject 93 different from the subject 92. Fig.25 In the figure, subjects 92 and 93 stand side by side, but subjects 92 and 93 may be located at distant positions. That is, the camera 84a and the camera 84b may be located at distant positions from each other, and may have a function of wirelessly transmitting a captured image to the information processing device 81b.
[0338] The subject 93 as well as the subject 92 wears the head mounted display 89. The head mounted display 89 displays the captured image Ia generated by the information processing device 81b in real time. Note that the head mounted display 89 worn by the subject 92 and the subject 93 has a function of outputting not only the captured image Ia but also the sound information output by the information processing device 81b. The subject 93 also wears the same controller 88 as the subject 92.
[0339] As described in the eighth embodiment, the information processing device 81b generates a captured image 1a (virtual image) in which the avatar 92a (first avatar) of the subject 92 is displayed. The information processing device 81b also displays the avatar 93a (second avatar) of the subject 93 in the captured image 1a. That is, the subject 92 and the subject 93 are the avatar 92a and the avatar 93a, respectively, and exist in the same virtual space of the captured image 1a.
[0340] The movement of the subject 92 is detected by the controller 88 and the head mounted display 89 and is sent to the information processing device 81b. Then, the movement of the subject 92 is reflected as the movement of the avatar 92a in the captured image 1a. Similarly, the movement of the subject 93 is detected by the controller 88 and the head mounted display 89 and is sent to the information processing device 81b. Then, the movement of the subject 93 is reflected as the movement of the avatar 93a in the captured image 1a. As described above, the subject 92 and the subject 93 move freely in the captured image 1a as the avatar 92a and the avatar 93a, respectively.
[0341] The information processing device 81b has a function that allows the avatar 92a to capture an image in a virtual space by using the virtual camera 84V. That is, when the subject 92 issues an instruction to operate the virtual camera 84V via the controller 88, the avatar 92a takes out the virtual camera 84V and captures an image in a direction in the virtual space as indicated by the subject 92. The captured image Ia is sent by the information processing device 81b to the head mounted display 89 worn by the subject 92 and the subject 93. Then, both the subject 92 and the subject 93 observe the captured image Ia in real time.
[0342] The information processing device 81b detects the sight line direction of the avatar 92a and the sight line direction of the avatar 93a appearing in the captured image Ia captured by the virtual camera 84V. When both the avatar 92a and the avatar 93a are looking at the virtual camera 84V, the information processing device 81b responds to the operation instruction from the subject 92 and takes a selfie of the captured image Ib when both the avatar 92a and the avatar 93a are looking at the virtual camera 84V.
[0343] The hardware configuration of the information processing device 81 b is obtained by adding a head mounted display 89 for a subject 93 and an imaging device 84 b and a controller 88 to the hardware configuration of the information processing device 81 a described in the eighth embodiment.
[0344] [9-2. Functional Configuration of Information Processing Device]
[0345] Fig.26 81 b is a functional block diagram showing an example of a functional configuration of an information processing device according to the ninth embodiment. The information processing device 81 b of the ninth embodiment includes an image capturing unit 51 , a subject feature detecting unit 53 , and a display control unit 54 .
[0346] The description of the image capture unit 51 will be omitted because the image capture unit 51 has the same function as that of the image capture unit of the eighth embodiment. The subject feature detection unit 53 has the same configuration as that of the subject feature detection unit 53 described in the eighth embodiment including the avatar detection unit 53h and the sight line direction detection unit 53b. However, the difference of this embodiment is that the avatar detection unit 53h detects the avatar 92a and the avatar 93a from the captured image Ia captured by the virtual camera 84V. The sight line direction detection unit 53b also detects the sight line direction E of the avatar 92a and the sight line direction F of the avatar 93a detected by the avatar detection unit 53h.
[0347] The display control unit 54 displays the avatar 92a and the avatar 93a while superimposing them on the captured image 1a.
[0348] [9-3. Flow of Processing Executed by Information Processing Device]
[0349] Fig. 27 1 is a flowchart showing an example of the flow of processing performed by the information processing device according to the ninth embodiment. Hereinafter, the flow of processing will be described in sequence.
[0350] Description of the flow of the processing in step S130 and step S131 will be omitted because the flow is the same as the flow of the processing in step S120 and step S121 described in the eighth embodiment.
[0351] Then, the avatar detection unit 53h detects the avatar 92a (first avatar) from the captured image Ia. Then, the sight line direction detection unit 53b detects the sight line direction E of the detected avatar 92a (step S132).
[0352] Then, the sight line direction detection unit 53b determines whether the avatar 92a is looking at the virtual camera 84V (step S133). In the case where it is determined that the avatar 92a is looking at the virtual camera (step S133: Yes), the process proceeds to step S135. Meanwhile, in the case where it is determined that the avatar 92a is not looking at the virtual camera (step S133: No), the process proceeds to step S134.
[0353] In the case where the determination result is "NO" in step S133, the display control unit 54 performs loop reproduction (live view display) of the avatar 93a in the captured image Ia (step S134). Thereafter, the process returns to step S132.
[0354] Meanwhile, in the case where the result of determination in step S133 is "yes", the avatar detection unit 53h detects the avatar 93a (second avatar) from the captured image 1a. Then, the sight line direction detection unit 53b detects the sight line direction F of the detected avatar 93a (step S135). Note that, in the case where it is determined that the avatar 92a is looking at the virtual camera, the information processing device 81b can issue an instruction of "take a photo together" to the subject 93 via the head mounted display 89.
[0355] Then, the sight line direction detection unit 53b determines whether the avatar 93a is looking at the virtual camera 84V (step S136). In the case where it is determined that the avatar 93a is looking at the virtual camera (step S136: Yes), the process proceeds to step S137. Meanwhile, in the case where it is determined that the avatar 93a is not looking at the virtual camera (step S136: No), the process returns to step S135.
[0356] In a case where the determination result is YES in step S136, the image capturing / video recording processing unit 51b takes a self-photograph while both the avatar 92a and the avatar 93a are looking at the virtual camera (step S137).
[0357] [9-4. Effects of the Ninth Embodiment]
[0358] As described above, according to the information processing device 81b of the ninth embodiment, the display control unit 54 (control unit) displays the avatar 93a (second avatar) of the subject 93 different from the subject 92 in the captured image Ia.
[0359] Therefore, the subject 92 (user) can cause another subject 93 to perform interactive actions or reactions in the virtual space.
[0360] Note that the effects described in this specification are merely examples and not limitations, and other effects may be exhibited. In addition, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications may be made without departing from the gist of the present disclosure.
[0361] For example, the present disclosure may also have the following configurations. (1)
[0363] An information processing device, comprising:
[0364] a control unit that controls the 3D model displayed in the captured image; and
[0365] a detection unit configured to detect features of a subject displayed in the captured image simultaneously with the 3D model, wherein:
[0366] The control unit changes a display mode of the 3D model according to the feature of the object detected by the detection unit. (2)
[0368] The information processing device according to (1), wherein:
[0369] The detection unit detects a sight line direction serving as a feature of the subject; and
[0370] The control unit changes the direction of the 3D model according to the detected sight line direction of the subject. (3)
[0372] The information processing device according to (2), wherein:
[0373] The control unit changes the direction of the 3D model by rotating the 3D model according to the line of sight direction of the subject. (4)
[0375] The information processing device according to (2), wherein:
[0376] The control unit changes the direction of the 3D model by rotating a coordinate system in which the 3D model is placed according to a line of sight direction of the subject. (5)
[0378] The information processing device according to any one of (1) to (4), wherein:
[0379] When the detection unit detects that the line of sight of the subject faces the direction in which the captured image is captured,
[0380] The control unit changes the direction of the 3D model to the direction in which the captured image was captured. (6)
[0382] The information processing device according to any one of (1) to (4), wherein:
[0383] When the detection unit detects that the line of sight of the subject is facing the 3D model,
[0384] The control unit changes the direction of the 3D model to face the subject. (7)
[0386] The information processing device according to any one of (1) to (6), wherein:
[0387] The detection unit detects a facial expression serving as a feature of the subject; and
[0388] In a case where it is determined that the facial expression of the subject is a smile, the control unit changes the direction of the 3D model to a direction in which the captured image is captured. (8)
[0390] The information processing device according to any one of (1) to (7), wherein:
[0391] The detection unit detects brightness serving as a feature of the subject; and
[0392] The control unit changes the brightness of the 3D model according to the brightness of the object. (9)
[0394] The information processing device according to any one of (1) to (8), wherein:
[0395] the detection unit detects a distance to the subject, the distance serving as a feature of the subject; and
[0396] The control unit changes the distance to the 3D model so that the 3D model and the subject are located within the depth of field of the captured image. (10)
[0398] The information processing device according to any one of (1) to (9), wherein:
[0399] the detecting unit detects a position of the subject in the captured image, the position serving as a feature of the subject; and
[0400] In a case where the position of the subject and the position of the 3D model are predicted to overlap, the control unit changes the display position of the 3D model to a position that does not overlap with the position of the subject. (11)
[0402] The information processing device according to any one of (1) to (10), wherein:
[0403] the detecting unit detects a position and a size of the subject in the captured image, the position and the size serving as features of the subject; and
[0404] The control unit adjusts the display position and size of the 3D model according to the position and size of the object. (12)
[0406] The information processing device according to any one of (1) to (11), further comprising:
[0407] An image capturing unit that captures an image of the subject and the 3D model whose display mode has been changed by the control unit. (13)
[0409] The information processing device according to (12), wherein:
[0410] The image capturing unit is a camera existing in the real world or a virtual camera existing in the virtual world. (14)
[0412] The information processing device according to any one of (1) to (13), wherein:
[0413] The subject is a first avatar that exists in the same virtual space as the 3D model and moves in the virtual space based on an operation from a user; and
[0414] The detection unit detects a feature of the first avatar displayed in a captured image captured by a second image capture unit existing in the virtual space. (15)
[0416] The information processing device according to (14), wherein:
[0417] The 3D model is a second avatar that moves in the virtual space based on an operation from a subject different from the subject. (16)
[0419] The information processing device according to any one of (1) to (14), wherein:
[0420] The 3D model is a model having 3D information about an object existing in the real world, the 3D information being generated by using a plurality of viewpoint images of the object captured from different viewpoints. (17)
[0422] The information processing device according to any one of (1) to (13), wherein:
[0423] In the case where a predetermined display is detected in the captured image,
[0424] The control unit displays the 3D model with the 3D model superimposed on the captured image. (18)
[0426] An information processing method, comprising:
[0427] a control process that controls the 3D model displayed in the captured image; and
[0428] a detection process for detecting features of a subject displayed in the captured image simultaneously with the 3D model, wherein:
[0429] The control process changes a display mode of the 3D model according to a feature of the object detected by the detection process. (19)
[0431] A program for causing a computer to:
[0432] a control unit that controls the 3D model displayed in the captured image; and
[0433] a detection unit configured to detect features of a subject displayed in the captured image simultaneously with the 3D model, wherein:
[0434] The program causes the control unit to change a display mode of the 3D model according to a feature of the object detected by the detection unit.
[0435] Reference numerals list
[0436] 51 Image capture unit
[0437] 52 AR marker detection unit
[0438] 53 Subject feature detection unit (detection unit)
[0439] 54 Display control unit (control unit)
[0440] 80 Mobile terminal (information processing device)
[0441] 81a, 81b Information processing device
[0442] 82 Display
[0443] 84, 84a, 84b Camera device (first image capturing unit)
[0444] 84V Virtual camera device (second image capture unit)
[0445] 85 AR mark (scheduled to be displayed)
[0446] 90 Subject (object)
[0447] 90M,90N,M 3D Model
[0448] 92 Subjects (Users)
[0449] 92a Incarnation (First Incarnation)
[0450] 93 Subject
[0451] 93a Incarnation (Second Incarnation)
[0452] Ia, Ib Capture image (virtual image)
[0453] E, F sight direction
[0454] I Capture Image
Claims
1. An information processing device, comprising: a control unit that controls a 3D model placed in a three-dimensional space and displayed in the captured captured image; as well as A detection unit, which detects sight line information of a subject displayed in the captured image simultaneously with the 3D model, wherein: The control unit changes the display mode of the 3D model according to the line of sight information of the subject detected by the detection unit, so that the direction corresponding to the line of sight of the 3D model approaches the line of sight information of the subject in a first display mode or approaches the direction in which the captured image is captured in a second display mode; The control unit changes the display mode of the 3D model between the first display mode and the second display mode based on whether the detected line of sight of the subject is facing the direction in which the captured image was captured or facing near the augmented reality marker of the 3D model.
2. The information processing device according to claim 1, wherein: The control unit changes the direction of the 3D model according to the detected line of sight information of the subject.
3. The information processing device according to claim 2, wherein: The control unit changes the direction of the 3D model by rotating the 3D model according to the line of sight information of the subject.
4. The information processing device according to claim 2, wherein: The control unit changes the direction of the 3D model by rotating a coordinate system in which the 3D model is placed according to the line of sight information of the subject.
5. The information processing device according to claim 2, wherein: When the detection unit detects that the line of sight information of the subject faces the direction in which the captured image is captured, The control unit changes the direction of the 3D model to the direction in which the captured image was captured.
6. The information processing device according to claim 2, wherein: When the detection unit detects that the line of sight of the subject is facing the 3D model, The control unit changes the direction of the 3D model to face the subject.
7. The information processing apparatus according to claim 1, wherein: The detection unit detects a facial expression serving as a feature of the subject; and In a case where it is determined that the facial expression of the subject is a smile, the control unit changes the direction of the 3D model to a direction in which the captured image is captured.
8. The information processing device according to claim 1, wherein: The detection unit detects brightness serving as a feature of the subject; and The control unit changes the brightness of the 3D model according to the brightness of the object.
9. The information processing device according to claim 1, wherein: the detection unit detects a distance to the subject, the distance serving as a feature of the subject; and The control unit changes the distance to the 3D model so that the 3D model and the subject are located within the depth of field of the captured image.
10. The information processing device according to claim 1, wherein: the detecting unit detects a position of the subject in the captured image, the position serving as a feature of the subject; and In a case where the position of the subject and the position of the 3D model are predicted to overlap, the control unit changes the display position of the 3D model to a position that does not overlap with the position of the subject.
11. The information processing apparatus according to claim 1, wherein: the detecting unit detects a position and a size of the subject in the captured image, the position and the size serving as features of the subject; and The control unit adjusts the display position and size of the 3D model according to the position and size of the object.
12. The information processing device according to claim 1, further comprising: An image capturing unit that captures an image of the subject and the 3D model whose display mode has been changed by the control unit.
13. The information processing device according to claim 12, wherein: The image capturing unit is a camera existing in the real world or a virtual camera existing in the virtual world.
14. The information processing apparatus according to claim 1, wherein: The subject is a first avatar that exists in the same virtual space as the 3D model and moves in the virtual space based on an operation from a user; and The detection unit detects a feature of the first avatar displayed in a captured image captured by a second image capture unit existing in the virtual space.
15. The information processing device according to claim 14, wherein: The 3D model is a second avatar that moves in the virtual space based on an operation from a subject different from the subject.
16. The information processing device according to claim 1, wherein: The 3D model is a model having 3D information about an object existing in the real world, the 3D information being generated by using a plurality of viewpoint images of the object captured from different viewpoints.
17. The information processing device according to claim 16, wherein: In the case where a predetermined display is detected in the captured image, The control unit superimposes the 3D model on the captured image to display the 3D model.
18. An information processing method, comprising: a control process that controls a 3D model placed in a three-dimensional space and displayed in the captured captured image; as well as Detection processing, which detects line of sight information of a subject displayed in the captured image simultaneously with the 3D model, wherein: The control process changes the display mode of the 3D model according to the line of sight information of the subject detected by the detection process, so that the direction corresponding to the line of sight of the 3D model approaches the line of sight information of the subject in a first display mode or approaches the direction in which the captured image is captured in a second display mode; The display mode of the 3D model is changed between the first display mode and the second display mode based on whether the detected line of sight of the subject is facing the direction in which the captured image was captured or facing the vicinity of the augmented reality marker of the 3D model.
19. A storage unit having stored therein a program for causing a computer to function as: a control unit that controls a 3D model placed in a three-dimensional space and displayed in the captured capture image; and A detection unit, which detects sight line information of a subject displayed in the captured image simultaneously with the 3D model, wherein: The program causes the control unit to change a display mode of the 3D model according to the line of sight information of the subject detected by the detection unit so that a direction corresponding to the line of sight of the 3D model approaches the line of sight information of the subject in a first display mode or approaches a direction in which the captured image is captured in a second display mode; The control unit changes the display mode of the 3D model between the first display mode and the second display mode based on whether the detected line of sight of the subject is facing the direction in which the captured image was captured or facing near the augmented reality marker of the 3D model.
Citation Information
Patent Citations
Encoding device and encoding method, and decoding device and decoding method
WO2017082076A1
Image processing device, three-dimensional image printing system, and image processing method
CN102630024A
System And Method For Augmented And Virtual Reality
CN104011788A
Image generating system and image processing method
CN107045711A
Information processing device, information processing system, and information processing method
CN108885799A
