Information processing device, information processing method, and program
The information processing device enables 3D models to react to viewer interactions by using a control unit to manage 3D models based on detected subject characteristics, addressing the lack of interaction in existing 3D object generation methods.
Patent Information
- Application Number
- JP2021526948
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-06-28
- Filing Date
- 2020-06-19
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2040-06-19
AI Technical Summary
Existing 3D object generation methods do not allow for interactive movements or reactions based on the observer's actions or state, lacking the ability for 3D objects to respond to viewer interactions.
An information processing device and method that includes a control unit to manage 3D models displayed in a captured image, with a detection unit to identify subject characteristics, allowing the 3D models to perform interactive actions and reactions based on the observer's movements or states.
Enables 3D models to interactively respond to viewer actions, maintaining realistic display quality and facilitating seamless integration with viewer movements and expressions.
Smart Images

Figure 0007582186000001 
Figure 0007582186000002 
Figure 0007582186000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that can cause a 3D object to perform an interactive action or reaction in response to the action of a viewer. [Background technology]
[0002] Conventionally, a method has been proposed in which a 3D object is generated within a viewing space using information sensed from a real 3D space, such as a multi-viewpoint image capturing a subject from different viewpoints, and an image (also called a volumetric video) in which the object appears to exist within the viewing space (for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] International Publication No. 2017 / 082076 Summary of the Invention [Problem to be solved by the invention]
[0004] The image generated by the above-mentioned method is a 3D shape image generated using an actual image of the subject, so an observer looking at the 3D shape image can experience a sense of immersion and realism as if the subject were closer to them.
[0005] However, it was not possible to make the generated 3D object move or react according to the observer's actions or state. In other words, the 3D object could not move or react interactively to the observer.
[0006] In view of this, the present disclosure proposes an information processing device, an information processing method, and a program that are capable of causing a 3D object displayed in a captured image including a subject, who is an observer, to perform an interactive action or react to the subject. [Means for solving the problem]
[0007] In order to solve the above problems, one form of information processing device according to the present disclosure includes a control unit that controls a 3D model displayed in a captured image, and a detection unit that detects characteristics of a subject displayed in the captured image simultaneously with the 3D model, and the control unit is an information processing device that changes the display mode of the 3D model depending on the characteristics of the subject detected by the detection unit. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a diagram showing an overview of the flow from imaging to generation of a 3D model. [Diagram 2] FIG. 1 is a diagram illustrating an AR application. [Diagram 3] FIG. 2 is a first diagram showing an example of an operation state of the mobile terminal according to the first embodiment. [Figure 4] FIG. 2 is a second diagram showing an example of an operating state of the mobile terminal according to the first embodiment. [Diagram 5] FIG. 2 is a hardware block diagram showing an example of a hardware configuration of a mobile terminal according to the first embodiment. [Figure 6] 2 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to the first embodiment. FIG. [Figure 7] 11A and 11B are diagrams illustrating a method for measuring a gaze direction. [Figure 8] 11A to 11C are diagrams illustrating a method for changing the drawing direction of a 3D model. [Figure 9] 5 is a flowchart showing an example of a flow of processing performed by the mobile terminal according to the first embodiment. [Figure 10] FIG. 11 is a diagram illustrating an example of an operation state of a mobile terminal according to a second embodiment. [Figure 11]10 is a flowchart showing an example of a flow of processing performed by a mobile terminal according to a second embodiment. [Figure 12] FIG. 11 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to a third embodiment. [Figure 13] 13 is a flowchart showing an example of a flow of processing performed by a mobile terminal according to the third embodiment. [Figure 14] FIG. 13 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to a fourth embodiment. [Figure 15] 13 is a flowchart showing an example of a flow of processing performed by a mobile terminal according to a fourth embodiment. [Figure 16] FIG. 13 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to a fifth embodiment. [Figure 17] 13 is a flowchart showing an example of a flow of processing performed by a mobile terminal according to the fifth embodiment. [Figure 18] FIG. 13 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to a sixth embodiment. [Figure 19] 20 is a flowchart showing an example of a flow of processing performed by a mobile terminal according to a sixth embodiment. [Figure 20] FIG. 13 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to the seventh embodiment. [Figure 21] 20 is a flowchart showing an example of a flow of processing performed by a mobile terminal according to the seventh embodiment. [Figure 22] FIG. 13 is a diagram illustrating an overview of an information processing device according to an eighth embodiment. [Figure 23] FIG. 23 is a functional block diagram showing an example of a functional configuration of an information processing device according to an eighth embodiment. [Figure 24] 20 is a flowchart showing an example of a flow of processing performed by an information processing device according to the eighth embodiment. [Diagram 25] FIG. 13 is a diagram illustrating an overview of an information processing device according to a ninth embodiment. [Figure 26] FIG. 13 is a functional block diagram showing an example of a functional configuration of an information processing device according to a ninth embodiment. [Figure 27]23 is a flowchart showing an example of a flow of processing performed by an information processing device according to the ninth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.
[0010] The present disclosure will be described in the following order. 1. First embodiment 1-1.Explanation of Prerequisites-3D Model 1-2.Explanation of Prerequisites - AR Application 1-3. Overview of mobile devices 1-4.Hardware configuration of mobile terminals 1-5. Functional configuration of mobile terminals 1-6. How to measure gaze direction 1-7.How to change the drawing direction of a 3D model 1-8. Processing flow of mobile devices 1-9. Effects of the First Embodiment 2. Second embodiment 2-1. Overview of the second embodiment 2-2. Processing flow performed by mobile devices 2-3. Advantages of the Second Embodiment 3. Third embodiment 3-1. Functional configuration of mobile terminal 3-2. Processing flow performed by mobile devices 3-3. Advantages of the Third Embodiment 4. Fourth embodiment 4-1. Functional configuration of mobile terminal 4-2. Processing flow performed by mobile devices 4-3. Advantages of the Fourth Embodiment 5. Fifth embodiment 5-1. Functional configuration of mobile terminal 5-2. Processing flow performed by mobile devices 5-3. Advantages of the Fifth Embodiment 6. Sixth embodiment 6-1. Functional configuration of mobile terminals 6-2. Processing flow performed by mobile devices 6-3. Effects of the Sixth Embodiment 7. Seventh embodiment 7-1. Functional configuration of mobile terminals 7-2. Processing flow of mobile devices 7-3. Effects of the Seventh Embodiment 8. Eighth embodiment 8-1. Overview of information processing device 8-2. Functional configuration of information processing device 8-3. Processing flow performed by information processing device 8-4. Effects of the Eighth Embodiment 9. Ninth embodiment 9-1. Overview of information processing device 9-2. Functional configuration of information processing device 9-3. Processing flow performed by information processing device 9-4. Effects of the 9th embodiment
[0011] 1. First Embodiment Before describing the embodiments of the present disclosure, prerequisites necessary for realizing the embodiments will be described.
[0012] [1-1. Explanation of Prerequisites - 3D Model] FIG. 1 is a diagram showing an overview of the process from capturing an image to generating a 3D model.
[0013] As shown in FIG. 1, generation of a 3D model includes capturing images of a subject 90 (object) using a plurality of imaging devices 1-1 to 1-3, and generating a 3D model 90M having 3D information of the subject 90 by 3D modeling.
[0014] Specifically, as shown in Fig. 1, a plurality of imaging devices 1-1 to 1-3 are arranged on the outside of a subject 90 existing in the real world, facing inward, so as to surround the subject 90. Fig. 1 shows an example in which the number of imaging devices is three, and the imaging devices 1-1 to 1-3 are arranged around the subject 90. In the example of Fig. 1, a person performing a predetermined action is regarded as the subject 90.
[0015] 3D modeling is performed using multiple viewpoint images volumetrically captured by three imaging devices 1-1 to 1-3 that are synchronously captured from different viewpoints, and a 3D model of subject 90 is generated for each image frame of the three imaging devices 1-1 to 1-3.
[0016] The 3D model is a model having 3D information of the subject 90. The 3D model 90M has mesh data that expresses the geometry information of the subject 90 as connections between vertices, called a polygon mesh, and texture information and depth information (distance information) corresponding to each polygon mesh. Note that the information possessed by the 3D model 90M is not limited to these, and other information may be possessed.
[0017] In addition, the 3D model 90M is subjected to so-called texture mapping, in which a texture that represents the color, pattern, or texture of the mesh is applied according to the mesh position. In the texture mapping, it is preferable to apply a texture that is view dependent in order to improve the reality of the 3D model 90M. However, since this increases the amount of calculation, a texture that is not dependent on the line of sight (view independent) may be applied.
[0018] The generated 3D model 90M is assigned an identification number according to the attributes (features) of the 3D model 90M. For example, if the 3D model 90M is a person, an identification number indicating the attribute value of each attribute item, such as the face direction, gaze direction, and facial expression, of the person, is assigned. The assigned identification numbers are collectively managed in a predetermined attribute table. Then, when an attribute value of a predetermined attribute item is given from outside, the attribute table is referenced and the 3D model 90M corresponding to the attribute value of the given attribute item is read out. The specific format of the attribute table is not important.
[0019] The content data including the read 3D model is transmitted to the playback device and played back. The content data including the 3D model is played back, and the 3D model is rendered, whereby a 3D shape image is displayed on the viewer's viewing device.
[0020] In the example of Fig. 1, a mobile terminal 80 such as a smartphone or a tablet terminal is used as the viewing device. A captured image I is displayed on a display screen 82 of the mobile terminal 80. A subject 92 captured by a camera 84 of the mobile terminal 80 is displayed in the captured image I. Furthermore, a 3D model 90M is superimposed on the captured image I by the action of an AR application (described later) that runs on the mobile terminal 80. This will be described in detail later.
[0021] [1-2. Explanation of Prerequisites - AR Application] FIG. 2 is a diagram illustrating the AR application.
[0022] The AR (Augmented Reality) application is an application that runs on the mobile terminal 80 and displays a 3D model 90M superimposed on the display screen 82 of the mobile terminal 80 when some trigger occurs.
[0023] Specifically, as shown in Fig. 2, when an AR marker 85, which is a predetermined display, is detected from a captured image I monitored by the mobile terminal 80, a 3D model 90M is displayed at a position corresponding to the AR marker 85. That is, the trigger is the detection of a predetermined display from the captured image I. The AR marker 85 may be in any form, such as a character string, a symbol string, a figure, a photograph, a barcode, a two-dimensional code, or the like, and is not limited to these forms. Furthermore, the AR marker 85 may be a 2D display or a 3D display.
[0024] Also, the trigger may be the detection of a predetermined target (a structure or a natural object) by performing a spatial analysis or the like on the captured image I monitored by the mobile terminal 80 on the display screen 82 (markerless). Furthermore, the trigger may be whether the current position acquired by the mobile terminal 80 using a GPS receiver is a predetermined position.
[0025] [1-3. Overview of mobile devices] An overview of a mobile terminal 80 according to this embodiment will be described with reference to FIG. 3. The mobile terminal 80 is an example of an information processing device in the present disclosure. FIG. 3 is a first diagram showing an example of an operating state of a mobile terminal according to the first embodiment. The mobile terminal 80 is, for example, a smartphone operated by a user. The mobile terminal 80 includes a display screen 82 that displays images and videos, and a camera 84 installed near the display screen 82 and facing in the same direction as the display screen 82.
[0026] The above-mentioned AR application is implemented in the mobile terminal 80. When the AR application is executed, the mobile terminal 80 displays the captured image I captured by the camera 18 on the display screen 82 in real time (so-called live view), and determines whether an AR marker 85 appears in the captured image I. If the AR marker 85 appears in the captured image I, the mobile terminal 80 displays the 3D model 90M superimposed on a position corresponding to the AR marker 85.
[0027] Furthermore, the mobile terminal 80 detects the characteristics of the subject 92 captured in the captured image I, and changes the display mode of the 3D model 90M according to the detected characteristics. It is assumed that a typical situation in which the mobile terminal 80 is used is a situation in which the user himself / herself becomes the subject 92 and takes a selfie with the camera 84 of the mobile terminal 80.
[0028] In this embodiment, the mobile terminal 80 detects the line of sight direction E of the subject 92 captured in the captured image I. Then, when the mobile terminal 80 determines that the line of sight direction E of the subject 92 is directed toward the camera 84, it orients the line of sight direction F of the 3D model 90M toward the camera 84.
[0029] When the line of sight direction E of the subject 92 and the line of sight direction F of the 3D model 90M are both directed toward the camera 84, the mobile terminal 80 captures a selfie image.
[0030] This allows the mobile terminal 80 to cause the 3D model 90M to move or react in accordance with the movement or state of the subject 92. Making the 3D model 90M react or undergo some state transition in this way is called 3D model AR with interaction.
[0031] 4 is a second diagram showing an example of an operating state of the mobile terminal according to the first embodiment. The mobile terminal 80 detects the line of sight direction E of the subject 92 appearing in the captured image I displayed on its display screen 82. When the detected line of sight direction E is directed toward the camera 84, the mobile terminal 80 selects a 3D model 90N having a line of sight direction F directed toward the camera 84, and displays the selected 3D model 90N superimposed on the captured image I.
[0032] The mobile terminal 80 then takes a selfie with both the subject 92 and the 3D model 90N looking at the camera. In this way, when the line of sight E, which is a characteristic of the subject 92, faces the direction of the camera 84, the display mode of the 3D model 90N can be changed so that the line of sight F of the 3D model 90N faces the direction of the camera 84.
[0033] [1-4.Hardware configuration of mobile terminals] Fig. 5 is a hardware block diagram showing an example of the hardware configuration of the mobile terminal according to the first embodiment. In particular, Fig. 4 shows only the elements related to this embodiment among the hardware components of the mobile terminal 80 of this embodiment. That is, the mobile terminal 80 has a configuration in which a CPU (Central Processing Unit) 32, a ROM (Read Only Memory) 34, a RAM (Random Access Memory) 36, a storage unit 38, and a communication interface 40 are connected by an internal bus 39.
[0034] The CPU 32 controls the overall operation of the mobile terminal 80 by expanding the control program P1 stored in the storage unit 38 or the ROM 34 on the RAM 36 and executing it. That is, the mobile terminal 80 has a general computer configuration operated by the control program P1. The control program P1 may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. The mobile terminal 80 may also execute a series of processes by hardware. The control program P1 executed by the CPU 32 may be a program in which processing is performed in chronological order according to the sequence described in this disclosure, or may be a program in which processing is performed in parallel or at a required timing such as when a call is made.
[0035] The storage unit 38 is configured, for example, with a flash memory, and stores information such as the control program P1 executed by the CPU 32 and the 3D model M. Since the 3D model M generally has a large capacity, it may be downloaded as necessary from an external server (not shown) connected to the mobile terminal 80 via the Internet or the like, and stored in the storage unit 38.
[0036] The communication interface 40 acquires operation information of the touch panel 14 laminated on the liquid crystal display 16 constituting the display screen 82 of the mobile terminal 80, via the touch panel interface 42. The communication interface 40 also displays image information and video information on the liquid crystal display 16, via the display interface 43. The communication interface 40 also controls the operation of the camera 84, via the camera interface 44.
[0037] Furthermore, the communication interface 40 communicates with an external server (not shown) by wireless communication or wired communication, allowing the mobile terminal 80 to receive a newly created 3D model M and transmit a generated captured image I to the outside.
[0038] [1-5. Functional configuration of mobile terminal] Fig. 6 is a functional block diagram showing an example of a functional configuration of a mobile terminal according to the first embodiment. The CPU 32 of the mobile terminal 80 deploys the control program P1 on the RAM 36 and runs it to realize an imaging unit 51, an AR marker detection unit 52, a subject feature detection unit 53, and a display control unit 54 shown in Fig. 6 as functional units.
[0039] Imaging unit 51 monitors video, captures images, and records videos using camera 84 included in mobile terminal 80. Imaging unit 51 further includes video monitor unit 51a and imaging / recording processing unit 51b. Camera 84 is an actual camera that exists in the real world, and is an example of a first imaging unit in the present disclosure.
[0040] The video monitor unit 51a performs a so-called live view, which displays the video captured by the camera 84 on the display screen 82 of the mobile terminal D1 in real time.
[0041] The imaging / recording processing section 51b stores the captured image I captured by the camera 84. The imaging / recording processing section 51b also performs selfie shooting.
[0042] The AR marker detection unit 52 detects an AR marker 85 from within the captured image I monitored by the processing of the video monitor unit 51a. Specifically, the AR marker detection unit 52 detects a predetermined AR marker 85 by performing template matching or the like.
[0043] The subject feature detection unit 53 detects a subject displayed simultaneously with the 3D model 90M in the captured image I monitored by the processing of the video monitor unit 51a. Then, the subject feature detection unit 53 detects the features of the detected subject 92. The subject feature detection unit 53 further includes a subject detection unit 53a and a gaze direction detection unit 53b. Note that the subject feature detection unit 53 is an example of a detection unit in the present disclosure.
[0044] The subject detection unit 53a detects the subject 92 displayed in the captured image I monitored by the imaging unit 51 simultaneously with the 3D model 90M.
[0045] Then, gaze direction detection section 53b detects the gaze direction E of subject 92 detected by subject detection section 53a. Note that gaze direction E is an example of a feature of subject 92 in the present disclosure.
[0046] The display control unit 54 controls the 3D model 90M displayed in the captured image I. More specifically, the display control unit 54 changes the display mode of the 3D model 90M according to the line of sight direction E of the subject 92 detected by the subject feature detection unit 53. In this embodiment, when the line of sight direction E of the subject 92 faces the camera 84, that is, when the subject 92 is looking at the camera, the display control unit 54 changes the line of sight direction F of the 3D model 90M to a direction facing the camera 84. That is, the line of sight direction F is an example of a display mode in the present disclosure. The display control unit 54 further includes a 3D model frame selection unit 54a and a 3D model drawing direction control unit 54b. The display control unit 54 is an example of a control unit in the present disclosure.
[0047] The 3D model frame selection unit 54a selects a default 3D model 90M determined in advance according to the AR marker 85 from among the multiple 3D models M stored in the storage unit 38. The 3D model frame selection unit 54a also selects a 3D model 90M having a line of sight direction F corresponding to the line of sight direction E of the subject 92 detected by the line of sight direction detection unit 53b from among the multiple 3D models M stored in the storage unit 38. More specifically, when the subject 92 is looking at the camera, the 3D model frame selection unit 54a selects a 3D model 90M in which the line of sight direction F of the 3D model 90M faces the camera 84.
[0048] The 3D model drawing direction control unit 54b displays the default 3D model 90M selected by the 3D model frame selection unit 54a superimposed on the captured image I displayed on the display screen 82 of the mobile terminal 80. The 3D model drawing direction control unit 54b also changes the orientation of the 3D model 90M by rotating the 3D model 90M. The 3D model drawing direction control unit 54b also changes the orientation of the 3D model 90M by rotating the coordinate system in which the 3D model 90M is placed, thereby generating a 3D model 90N. The 3D model drawing direction control unit 54b also displays the 3D model 90N whose orientation has been changed superimposed on the captured image I displayed on the display screen 82 of the mobile terminal 80.
[0049] [1-6. How to measure gaze direction] 7 is a diagram for explaining a method for measuring the gaze direction. Various methods have been proposed for detecting the gaze direction E from a face image of the subject 92, and any of these methods may be used in this embodiment.
[0050] Fig. 7 explains three typical methods for measuring the gaze direction. Fig. 7A shows a method for detecting the gaze direction E by capturing an image of the face of subject 92 with camera 84, detecting the eyeball area from the captured face image, and detecting the gaze direction E based on the shape characteristics of the detected eyeball.
[0051] 7A shows an enlarged view of the left eye of subject 92 detected from captured image I captured by camera 84. The distance dp between the center of pupil P and the inner corner Q of the eye changes depending on the gaze direction. Specifically, as shown in FIG. 7A, when the gaze turns to the left (toward the outer corner S of the eye), the distance dp becomes larger. On the other hand, when the gaze turns to the right (toward the inner corner Q of the eye), the distance dp becomes smaller. Therefore, by measuring the distance dp, the approximate gaze direction can be calculated.
[0052] 7B shows an example in which the face of subject 92 is imaged by camera 84 while being illuminated by infrared LED 86. It is assumed that camera 84 has sensitivity to light of the wavelength emitted by infrared LED 86. The gaze measurement method shown in FIG. 7B detects the area of the eyeball from the captured face image, and detects gaze direction E based on the position of the reflected image (corneal reflection image) of infrared LED 86 on the cornea of the detected eyeball. It is to be noted that the wavelength of light emitted by infrared LED 86 is invisible to the naked eye, so that the gaze direction can be detected without interfering with subject 92.
[0053] Fig. 7B shows an enlarged view of the left eye of subject 92 detected from captured image I captured by camera 84. The positional relationship between the center of pupil P and corneal reflection image R changes depending on the gaze direction. Specifically, as shown in Fig. 7B, when the gaze is directed to the left (toward the outer corner S of the eye), the pupil P is located on the outer corner S side of the corneal reflection image R. On the other hand, when the gaze is directed to the right (toward the inner corner Q of the eye), the pupil P is located on the inner corner Q side of the corneal reflection image R. Therefore, by measuring the positional relationship between the pupil P and corneal reflection image R, the approximate gaze direction can be detected.
[0054] In Fig. 7C, the face of subject 92 is captured by camera 84, and the gaze direction is detected based on the orientation of the captured face. In general, when humans change their gaze direction quickly or significantly, they change the gaze direction by moving their head without moving their eyeballs. Therefore, it is often the case that the facial orientation and gaze direction are considered to be approximately the same. When this approximation holds, the gaze direction can be detected by the method shown in Fig. 7C.
[0055] As shown in the left part of Figure 7C, when subject 92 is facing forward, the distance a1 between a center line C1 passing vertically through the face of subject 92 and a straight line C2 passing through the right eye and parallel to center line C1, and the distance a2 between center line C1 and a straight line C3 passing through the left eye and parallel to center line C1, are approximately equal.
[0056] When the subject 92 faces to the right as shown in the center of Fig. 7C, the distance a1 is greater than the distance a2. When the subject 92 faces to the left as shown in the right of Fig. 7C, the distance a1 is smaller than the distance a2.
[0057] In this manner, by comparing the magnitude relationship between the distance a1 and the distance a2, the approximate direction of the line of sight can be detected.
[0058] In addition, the shape and movement of the eyeball vary from person to person. Therefore, in order to stably detect the eyeball of the subject 92, a method that uses a highly robust eyeball model created by using a technique such as machine learning has been recently used.
[0059] [1-7. How to change the drawing direction of a 3D model] 8 is a diagram for explaining a method for changing the drawing direction of a 3D model. Since the 3D model 90M has directionality, when the 3D model 90M is drawn on the display screen 82, the 3D model 90M is drawn in a default direction, for example, in the positive direction of the Z axis. In this embodiment, the orientation of the 3D model 90M is changed according to the line of sight direction E, which is a characteristic of the subject 92, and the 3D model 90M is drawn as a 3D model 90N. Therefore, in this embodiment, two types of drawing direction control methods described below are used in combination.
[0060] The first method is to rotate 3D model 90M itself in a desired direction to create 3D model 90N, as shown in FIG. 8A.
[0061] The second method is to rotate the coordinate system XYZ in which the 3D model 90M is placed in a desired direction, and render the 3D model 90N in the rotated coordinate system, as shown in FIG. 8B.
[0062] In this embodiment, either method may be used, but the method of rotating the coordinate system XYZ in which the 3D model 90M is placed requires less calculation, but if the 3D model 90M has a texture that does not depend on the viewpoint position, the reality of the 3D model 90M may decrease. On the other hand, the method of rotating the 3D model 90M itself requires more calculation, but if the 3D model 90M has a texture that depends on the line of sight, the reality of the 3D model 90M is maintained. Therefore, it is desirable to change the drawing direction after giving sufficient consideration to which method to apply.
[0063] For example, when the amount of rotation of the 3D model 90M is small, it is desirable to apply a method of rotating a coordinate system, which is considered to cause little loss in the reality of the 3D model 90M due to the rotation. On the other hand, when the amount of rotation of the 3D model 90M is large, it is desirable to apply a method of rotating the 3D model 90M itself in order to maintain the reality of the 3D model 90M.
[0064] [1-8. Processing flow of mobile devices] 9 is a flowchart showing an example of the flow of processing performed by the mobile terminal according to the first embodiment. The flow of processing will be explained step by step below.
[0065] The video monitor unit 51a activates the camera 84 (step S10). After that, the camera 84 goes into a live view state in which the captured image I is monitored in real time.
[0066] The AR marker detection unit 52 detects the AR marker 85 from within the captured image I (step S11).
[0067] Next, the AR marker detection unit 52 determines whether the AR marker 85 has been detected (step S12). If it is determined that the AR marker 85 has been detected (step S12: Yes), the process proceeds to step S13. On the other hand, if it is determined that the AR marker 85 has not been detected (step S12: No), the process returns to step S11.
[0068] If the result of the determination in step S12 is Yes, the 3D model frame selection unit 54a selects a default 3D model 90M, which is determined in advance according to the AR marker 85, from among the 3D models M stored in the storage unit 38. Then, the 3D model drawing direction control unit 54b displays the selected 3D model 90M by superimposing it on the captured image I at a position according to the AR marker 85 (step S13).
[0069] Next, the subject detection section 53a detects the subject 92 from within the captured image I (step S14).
[0070] The subject detection unit 53a determines whether the subject 92 has been detected (step S15). If it is determined that the subject 92 has been detected (step S15: Yes), the process proceeds to step S16. On the other hand, if it is determined that the subject 92 has not been detected (step S15: No), the process returns to step S14.
[0071] If the answer to step S15 is Yes, gaze direction detection section 53b detects the gaze direction of subject 92 (step S16).
[0072] Next, gaze direction detection unit 53b determines whether subject 92 is looking at the camera, that is, whether gaze direction E of subject 92 is facing the direction of camera 84 (step S17). If it is determined that gaze direction E of subject 92 is facing the direction of camera 84 (step S17: Yes), the process proceeds to step S19. On the other hand, if it is not determined that gaze direction E of subject 92 is facing the direction of camera 84 (step S17: No), the process proceeds to step S18.
[0073] If the result of the determination in step S17 is No, the 3D model frame selection unit 54a selects video frames of the 3D model 90M from among the 3D models M stored in the storage unit 38. Then, the 3D model drawing direction control unit 54b loop-plays the video frames selected by the 3D model frame selection unit 54a (step S18). After that, the process returns to step S16.
[0074] On the other hand, if the result of the determination in step S17 is Yes, the 3D model drawing direction control unit 54b orients the 3D model 90M toward the camera 84 (step S19). Note that the 3D model drawing direction control unit 54b may set the orientation of the 3D model 90M by rotating the direction of the coordinate system in which the 3D model 90M is drawn, or may set the orientation by rotating the 3D model 90M itself.
[0075] Next, the 3D model drawing direction control unit 54b draws the 3D model 90N selected by the 3D model frame selection unit 54a, with eyes open and line of sight F facing the camera 84, at the drawing position set in step S19 (step S20).
[0076] Then, the imaging / recording processing unit 51b takes a selfie shot with both the subject 92 and the 3D model 90N looking at the camera (step S21).
[0077] [1-9. Effects of the first embodiment] As described above, according to the mobile terminal 80 of the first embodiment, the subject feature detection unit 53 (detection unit) detects the line of sight E (feature) of the subject 92 displayed simultaneously with the 3D model 90M in the captured image I. Then, the display control unit 54 (control unit) changes the line of sight F (display mode) of the 3D model 90M according to the line of sight E of the subject 92 detected by the subject feature detection unit 53 so that the 3D model 90M faces the camera 84.
[0078] This makes it possible to make the 3D model 90M displayed in the captured image I including the subject 92 perform interactive movements and reactions, such as looking at the camera together with the subject 92.
[0079] Furthermore, according to the mobile terminal 80 of the first embodiment, the subject feature detection unit 53 (detection unit) detects the gaze direction E, which is a characteristic of the subject 92, and the display control unit 54 (control unit) changes the gaze direction F (face direction) of the 3D model 90M according to the detected gaze direction E of the subject 92.
[0080] This allows the 3D model 90M to perform interactive actions and reactions according to the line of sight E of the subject 92.
[0081] Furthermore, according to the mobile terminal 80 of the first embodiment, the display control unit 54 (control unit) rotates the 3D model 90M in accordance with the line of sight E of the subject 92, thereby changing the orientation of the 3D model 90M.
[0082] This makes it possible to maintain the reality of the 3D model 90M having a texture according to the viewpoint position even when the 3D model 90M is rotated.
[0083] Furthermore, according to the mobile terminal 80 of the first embodiment, the display control unit 54 (control unit) changes the orientation of the 3D model 90M by rotating the coordinate system in which the 3D model 90M is placed, in accordance with the line of sight direction E of the subject 92.
[0084] This allows the process of rotating a 90M 3D model to be performed quickly with a small amount of calculations.
[0085] Furthermore, according to the mobile terminal 80 of the first embodiment, when the subject feature detection unit 53 (detection unit) detects that the gaze direction E of the subject 92 is facing the direction in which the captured image I was captured, the display control unit 54 (control unit) changes the orientation of the 3D model 90M to the direction in which the captured image I was captured.
[0086] This allows the 3D model 90M to perform interactive actions and reactions in accordance with the subject 92.
[0087] Furthermore, in the mobile terminal 80 of the first embodiment, the imaging unit 51 captures an image of the 3D model 90M and the subject 92 after the display control unit 54 (control unit) has changed the display mode.
[0088] This allows, for example, a selfie to be taken between the subject 92 and the 3D model 90M by the camera 84.
[0089] In addition, in the mobile terminal 80 of the first embodiment, the imaging unit 51 is a camera 84 that exists in the real world, or a virtual camera 84V (see the eighth embodiment) that exists in the virtual world.
[0090] This allows a selfie to be taken between the subject 92 and the 3D model 90M by the camera 84. Furthermore, as will be described in detail in the eighth embodiment, a selfie can be taken between the avatar 92a, which is an alter-ego of the subject 92, and the 3D model 90M by the virtual camera 84V.
[0091] Furthermore, according to the mobile terminal 80 of the first embodiment, the 3D model 90M is a model having 3D information of a subject 90 (object) existing in the real world, generated using multiple viewpoint images captured from different viewpoints.
[0092] This makes it possible to generate a 3D model 90M that can be observed with the same quality from a plurality of different directions. Therefore, the orientation of the 3D model 90M displayed by being superimposed on the captured image I can be freely changed while maintaining the display quality.
[0093] Furthermore, according to the mobile terminal 80 of the first embodiment, when an AR marker 85 (predetermined display) is displayed in the captured image I, the display control unit 54 (control unit) displays a 3D model 90M superimposed on the captured image I.
[0094] This allows the display of 90M of a 3D model to be performed easily and quickly without complicated operations or procedures.
[0095] 2. Second Embodiment The second embodiment of the present disclosure is an example of a mobile terminal 80 (information processing device) equipped with a function that allows a subject 92 and a 3D model 90M to make eye contact and gaze at each other.
[0096] The hardware configuration and functional configuration of the mobile terminal 80 of this embodiment are the same as those of the mobile terminal 80 described in the first embodiment, and therefore a description thereof will be omitted. However, in order to realize the unique functions of this embodiment, the control program stored in the storage unit 38 uses a program different from the control program P1 (see FIG. 5).
[0097] [2-1. Overview of the second embodiment] 10 is a diagram showing an example of an operating state of a mobile terminal according to the second embodiment. The mobile terminal 80 detects a line of sight E of a subject 92 appearing in a captured image I displayed on its display screen 82. When the detected line of sight E is directed toward the vicinity of the AR marker 85, the mobile terminal 80 selects a 3D model 90N having a line of sight F directed toward the subject 92, and displays the 3D model 90N superimposed on the captured image I.
[0098] Then, the mobile terminal 80 takes a selfie shot in a state where the subject 92 and the 3D model 90N make eye contact. In this way, when the eye gaze direction E, which is a characteristic of the subject 92, faces the vicinity of the AR marker 85, the display mode of the 3D model 90N can be changed so that the eye gaze direction F of the 3D model 90N faces the direction of the subject 92.
[0099] [2-2. Processing flow performed by mobile devices] 11 is a flowchart showing an example of the flow of processing performed by the mobile terminal according to the first embodiment. The flow of the processing will be explained step by step below.
[0100] The process flow from step S30 to step S36 in FIG. 11 is the same as the process flow from step S10 to step S16 in FIG. 9 described in the first embodiment, and therefore a description thereof will be omitted.
[0101] In step S36, after the gaze direction detection unit 53b detects the gaze direction of the subject 92, the gaze direction detection unit 53b determines whether the subject 92 is facing the direction of the AR marker 85 (step S37). If it is determined that the gaze direction of the subject 92 is facing the direction of the AR marker 85 (step S37: Yes), the process proceeds to step S39. On the other hand, if it is not determined that the gaze direction of the subject 92 is facing the direction of the AR marker 85 (step S37: No), the process proceeds to step S38.
[0102] If the result of the determination in step S37 is No, the 3D model frame selection unit 54a selects video frames of the 3D model 90M from among the 3D models M stored in the storage unit 38. Then, the 3D model drawing direction control unit 54b loop-plays the video frames selected by the 3D model frame selection unit 54a (step S38). After that, the process returns to step S36.
[0103] On the other hand, if the result of the determination in step S37 is Yes, the 3D model drawing direction control unit 54b orients the drawing position of the 3D model 90M toward the face direction of the subject 92 (step S39).
[0104] Next, the 3D model drawing direction control unit 54b draws the 3D model 90N selected by the 3D model frame selection unit 54a, with its eyes open and its line of sight facing toward the face of the subject 92, at the drawing position set in step S39 (step S40).
[0105] Then, the imaging / recording processing unit 51b performs a selfie shot in a state in which the subject 92 and the 3D model 90N are aligned in line of sight (step S41).
[0106] [2-3. Effects of the second embodiment] As described above, according to the mobile terminal 80 of the second embodiment, when the subject feature detection unit 53 (detection unit) detects that the gaze direction E of the subject 92 is facing the direction of the 3D model 90M, the display control unit 54 (control unit) renders the 3D model 90N with its orientation changed to face the subject 92.
[0107] This allows selfie photography to be performed with subject 92 and 3D model 90N meeting each other's gazes.
[0108] 3. Third embodiment The third embodiment of the present disclosure is an example of a mobile terminal 80 (information processing device) having a function of taking a selfie photograph by pointing a 3D model 90M in a smiling state toward the camera 84 when a subject 92 is smiling and looking at the camera 84. That is, the mobile terminal 80 of this embodiment changes the display mode (gaze direction and facial expression) of the 3D model 90M according to the gaze direction and facial expression, which are characteristics of the subject 92.
[0109] The hardware configuration of the mobile terminal 80 of this embodiment is the same as that of the mobile terminal 80 described in the first embodiment, and therefore a description thereof will be omitted. Note that, in order to realize the unique functions of this embodiment, the control program stored in the storage unit 38 uses a program different from the control program P1 (see FIG. 5).
[0110] [3-1. Functional configuration of mobile terminal] Fig. 12 is a functional block diagram showing an example of the functional configuration of a mobile terminal according to the third embodiment. A mobile terminal 80 of the third embodiment has a functional configuration in which some functions are added to the mobile terminal 80 of the first embodiment (see Fig. 6). Only the differences in the functional configuration from the mobile terminal 80 of the first embodiment will be described below.
[0111] That is, subject feature detection section 53 includes smile detection section 53c in addition to subject detection section 53a and gaze direction detection section 53b.
[0112] Smile detection unit 53c detects whether subject 92 captured in captured image I is smiling. Smile detection unit 53c extracts the facial area of subject 92 detected by subject detection unit 53a. Smile detection unit 53c then compares the extracted facial area with a sample indicating a smile among a large number of facial expression samples stored in an expression database (not shown in FIG. 12). Smile detection unit 53c determines that subject 92 is smiling when the extracted facial area shows a high degree of comparison with the sample indicating a smile.
[0113] The smile detection unit 53c may use other image processing algorithms to determine whether or not a smile is occurring.
[0114] [3-2. Processing flow of mobile devices] 13 is a flowchart showing an example of the flow of processing performed by the mobile terminal according to the third embodiment. The flow of processing will be explained step by step below.
[0115] The process flow from step S50 to step S55 in FIG. 13 is the same as the process flow from step S10 to step S15 in FIG. 9 described in the first embodiment, and therefore description thereof will be omitted.
[0116] In step S55, if it is determined that the subject 92 has been detected (step S55: Yes), the gaze direction detection section 53b detects the gaze direction E of the subject 92. Then, the smile detection section 53c determines whether the subject 92 is smiling (step S56).
[0117] Next, if it is determined in step S56 that the subject 92 is smiling, gaze direction detection unit 53b determines whether the gaze direction E of the subject 92 is facing the direction of the camera 84 (step S57). If it is determined that the subject 92 is smiling and the gaze direction E is facing the direction of the camera 84 (step S57: Yes), the process proceeds to step S59. On the other hand, if it is not determined that the subject 92 is smiling and the gaze direction E is facing the direction of the camera 84 (step S57: No), the process proceeds to step S58.
[0118] If the result of the determination in step S57 is No, the 3D model frame selection unit 54a selects a video frame of the 3D model 90M from the 3D models M stored in the storage unit 38. Then, the 3D model drawing direction control unit 54b loop-plays the video frame selected by the 3D model frame selection unit 54a (step S58). After that, the process returns to step S56.
[0119] On the other hand, if the result of the determination in step S57 is Yes, the 3D model drawing direction control unit 54b orients the drawing position of the 3D model 90M in the direction of the camera 84 (step S59).
[0120] Next, the 3D model drawing direction control unit 54b draws the 3D model 90N selected by the 3D model frame selection unit 54a, which has its eyes open, its line of sight facing the camera 84, and is smiling, at the drawing position set in step S59 (step S60).
[0121] Then, the imaging / recording processing unit 51b takes a selfie shot with both the subject 92 and the 3D model 90N looking at the camera (step S61).
[0122] [3-3. Effects of the third embodiment] As described above, according to the mobile terminal 80 of the third embodiment, the subject characteristic detection unit 53 (detection unit) detects a smile (facial expression), which is a characteristic of the subject 92. Then, when it is determined that the facial expression of the subject 92 is a smile, the display control unit 54 (control unit) changes the orientation of the 3D model 90M to the direction of the camera 84, that is, the direction in which the captured image I was captured.
[0123] This allows a selfie to be taken while both the subject 92 and the 3D model 90N are smiling and looking at the camera.
[0124] 4. Fourth Embodiment The fourth embodiment of the present disclosure is an example of a mobile terminal 80 (information processing device) having a function for taking a selfie by automatically correcting the brightness and skin color of a 3D model 90M according to the shooting environment of a subject 92. That is, the mobile terminal 80 of this embodiment changes the display mode (brightness) of the 3D model 90M according to brightness (e.g., skin brightness), which is a feature of the subject 92.
[0125] The hardware configuration of the mobile terminal 80 of this embodiment is the same as that of the mobile terminal 80 described in the first embodiment, and therefore a description thereof will be omitted. Note that, in order to realize the unique functions of this embodiment, the control program stored in the storage unit 38 uses a program different from the control program P1 (see FIG. 5).
[0126] [4-1. Functional configuration of mobile terminal] Fig. 14 is a functional block diagram showing an example of the functional configuration of a mobile terminal according to the fourth embodiment. A mobile terminal 80 of the fourth embodiment has a functional configuration in which some functions are exchanged with those of the mobile terminal 80 of the first embodiment (see Fig. 6). Only the differences in the functional configuration from the mobile terminal 80 of the first embodiment will be described below.
[0127] The subject feature detection section 53 includes a photographing environment evaluation section 53d instead of the gaze direction detection section 53b.
[0128] The photographing environment evaluation unit 53d measures the brightness of the photographing environment of the captured image I and the brightness of the subject 92 captured in the captured image I. For example, the photographing environment evaluation unit 53d divides the captured image I into an area of the subject 92 and an area other than the subject 92, and obtains the average brightness of each area. The photographing environment evaluation unit 53d may also measure the skin tone of the subject 92.
[0129] Moreover, the display control unit 54 includes a 3D model brightness control unit 54c instead of the 3D model drawing direction control unit 54b.
[0130] The 3D model brightness control unit 54c corrects the brightness of the texture of the 3D model 90M. For example, the 3D model brightness control unit 54c compares the brightness of the shooting environment measured by the shooting environment evaluation unit 53d with the brightness of the environment when the 3D model 90M was generated. If the brightness of the environment when the 3D model 90M was generated is brighter than the brightness of the shooting environment measured by the shooting environment evaluation unit 53d, the texture of the 3D model 90M is corrected to be darker. On the other hand, if the brightness of the environment when the 3D model 90M was generated is darker than the brightness of the shooting environment measured by the shooting environment evaluation unit 53d, the texture of the 3D model 90M is corrected to be brighter.
[0131] In addition, the 3D model brightness control unit 54c may correct the texture of the 3D model 90M based on the skin brightness and skin tone of the subject 92 captured in the captured image I so that the skin brightness and skin tone of the 3D model 90M are approximately the same as the skin brightness and skin tone of the subject 92.
[0132] [4-2. Processing flow of mobile devices] 15 is a flowchart showing an example of the flow of processing performed by a mobile terminal according to the fourth embodiment. The flow of processing will be explained step by step below.
[0133] The process flow from step S70 to step S73 in FIG. 15 is the same as the process flow from step S10 to step S13 in FIG. 9 described in the first embodiment, and therefore will not be described.
[0134] Following step S73, the subject detection unit 53a detects the subject 92 from within the captured image I. Then, the shooting environment evaluation unit 53d detects the brightness of the shooting environment of the captured image I and the brightness of the subject 92 captured in the captured image I (step S74).
[0135] Next, the photographing environment evaluation unit 53d judges whether the subject 92 is captured in the captured image I and whether the brightness of the photographing environment is brighter than the photographing environment of the 3D model 90M (step S75). If it is judged that the condition is satisfied (step S75: Yes), the process proceeds to step S76. On the other hand, if it is not judged that the condition is satisfied (step S75: No), the process proceeds to step S77.
[0136] If the result of the determination in step S75 is Yes, the 3D model brightness control unit 54c corrects the texture of the 3D model 90M to be brighter (step S76). After that, the process proceeds to step S78.
[0137] On the other hand, if the result of the determination in step S75 is No, the 3D model brightness control unit 54c corrects the texture of the 3D model 90M to be darker (step S77).
[0138] Following step S76 or step S77, the 3D model drawing direction control unit 54b draws, in the captured image I, a 3D model 90N in which the texture brightness correction corresponding to step S76 or step S77 has been performed on the 3D model 90M selected by the 3D model frame selection unit 54a (step S78).
[0139] Then, the imaging / recording processing unit 51b performs selfie shooting in a state in which both the subject 92 and the 3D model 90N are captured (step S79).
[0140] [4-3. Effects of the Fourth Embodiment] As described above, according to the mobile terminal 80 of the fourth embodiment, the subject characteristic detection unit 53 (detection unit) detects brightness, which is a characteristic of the subject 92. Then, the display control unit 54 (control unit) changes the brightness of the 3D model 90M in accordance with the brightness of the subject 92.
[0141] This allows a selfie to be taken with the 3D model 90N, the texture brightness of which has been corrected to match the facial tone of the subject 92.
[0142] 5. Fifth Embodiment The fifth embodiment of the present disclosure is an example of a mobile terminal 80 (information processing device) having a function of correcting the rendering position (distance from the camera 84) of the 3D model 90M according to the position (distance from the camera 84) of the subject 92 and taking a selfie with both the subject 92 and the 3D model 90M in focus. That is, the mobile terminal 80 of this embodiment changes the display mode (rendering position) of the 3D model 90M according to the position (distance from the camera 84), which is a characteristic of the subject 92.
[0143] The hardware configuration of the mobile terminal 80 of this embodiment is the same as that of the mobile terminal 80 described in the first embodiment, and therefore a description thereof will be omitted. Note that, in order to realize the unique functions of this embodiment, the control program stored in the storage unit 38 uses a program different from the control program P1 (see FIG. 5).
[0144] [5-1. Functional configuration of mobile terminal] Fig. 16 is a functional block diagram showing an example of the functional configuration of a mobile terminal according to the fifth embodiment. A mobile terminal 80 according to the fifth embodiment has a functional configuration in which some functions are exchanged with those of the mobile terminal 80 according to the first embodiment (see Fig. 6). Only the differences in the functional configuration from the mobile terminal 80 according to the first embodiment will be described below.
[0145] The subject feature detection section 53 includes a depth of field calculation section 53e instead of the gaze direction detection section 53b.
[0146] The depth of field calculation unit 53e calculates the distance from the camera 84 to the subject 92 and the AR marker 85 captured in the captured image I. The depth of field calculation unit 53e also calculates the depth of field of the captured image I. The depth of field calculation unit 53e calculates the depth of field from the aperture (F-number) of the camera 84, the focal length of the lens, and the distance to the subject 92.
[0147] Moreover, the display control unit 54 includes a 3D model drawing distance control unit 54d instead of the 3D model drawing direction control unit 54b.
[0148] The 3D model rendering distance control unit 54d controls the rendering position of the 3D model 90M (the distance from the camera 84 to the position where the 3D model 90M is rendered). More specifically, the 3D model rendering distance control unit 54d controls the rendering position of the 3D model 90M so that the 3D model 90M falls within the same depth of field as the subject 92.
[0149] [5-2. Processing flow performed by mobile devices] 17 is a flowchart showing an example of the flow of processing performed by a mobile terminal according to the fifth embodiment. The flow of processing will be explained step by step below.
[0150] The process flow from step S80 to step S83 in FIG. 17 is the same as the process flow from step S10 to step S13 in FIG. 9 described in the first embodiment, and therefore description thereof will be omitted.
[0151] Following step S83, the depth of field calculation unit 53e calculates the distance between the camera 84 and the AR marker 85 shown in the captured image I (step S84).
[0152] Next, the subject detection unit 53a detects the subject 92 from within the captured image I (step S85). At this time, the depth of field calculation unit 53e calculates the distance between the camera 84 and the subject 92.
[0153] The depth of field calculation unit 53e determines whether the subject 92 is captured in the captured image I and whether the AR marker 85 and the subject 92 are within the same depth of field (step S86). If it is determined that the condition is satisfied (step S86: Yes), the process proceeds to step S88. On the other hand, if it is not determined that the condition is satisfied (step S86: No), the process proceeds to step S87. Note that whether the AR marker 85 and the subject 92 are within the same depth of field may be determined based on whether the distance between the AR marker 85 and the subject 92, which is calculated based on the results of steps S84 and S85, falls within the depth of field calculated from the imaging conditions of the subject 92.
[0154] If step S86 returns No, that is, if the AR marker 85 and the subject 92 are not within the same depth of field, the 3D model drawing distance control unit 54d draws the drawing position of the 3D model 90M in the captured image I at a distance that is within the same depth of field as the subject 92 (step S87).
[0155] Next, the 3D model drawing direction control unit 54b draws the 3D model 90M selected by the 3D model frame selection unit 54a at the drawing position set in step S87 (step S88).
[0156] Then, the imaging / recording processing unit 51b captures a selfie in a state in which the subject 92 and the 3D model 90M are within the same depth of field (step S89).
[0157] [5-3. Effects of the Fifth Embodiment] As described above, according to the mobile terminal 80 of the fifth embodiment, the subject characteristic detection unit 53 (detection unit) detects the distance from the camera 84 to the subject 92, which is a characteristic of the subject 92. Then, the display control unit 54 (control unit) changes the distance to the 3D model 90M so that the 3D model 90M and the subject 92 are within the depth of field of the captured image I.
[0158] This allows a selfie to be taken with both the subject 92 and the 3D model 90M in focus.
[0159] 6. Sixth Embodiment The sixth embodiment of the present disclosure is an example of a mobile terminal 80 (information processing device) having a function of correcting the rendering position of a 3D model 90M according to the position of a subject 92 in a captured image I, and taking a selfie without overlapping the subject 92 and the 3D model 90M. That is, the mobile terminal 80 of this embodiment changes the display mode (rendering position) of the 3D model 90M according to the position, which is a characteristic of the subject 92.
[0160] The hardware configuration of the mobile terminal 80 of this embodiment is the same as that of the mobile terminal 80 described in the first embodiment, and therefore a description thereof will be omitted. Note that, in order to realize the unique functions of this embodiment, the control program stored in the storage unit 38 uses a program different from the control program P1 (see FIG. 5).
[0161] [6-1. Functional configuration of mobile terminal] Fig. 18 is a functional block diagram showing an example of the functional configuration of a mobile terminal according to the sixth embodiment. A mobile terminal 80 according to the sixth embodiment has a functional configuration in which some functions are exchanged with those of the mobile terminal 80 according to the first embodiment (see Fig. 6). Only the differences in the functional configuration from the mobile terminal 80 according to the first embodiment will be described below.
[0162] The subject feature detection section 53 includes an overlap determination section 53f instead of the gaze direction detection section 53b.
[0163] The overlap determination unit 53f determines whether the subject 92 and the AR marker 85 overlap in the captured image I. Specifically, the overlap determination unit 53f detects the position of the subject 92 in the captured image I. Then, the overlap determination unit 53f compares the detected position of the subject 92 with the position of the AR marker 85 detected by the AR marker detection unit 52. If the subject 92 and the AR marker 85 overlap, the overlap determination unit 53f predicts that the subject 92 and the 3D model 90M will overlap when the 3D model 90M is drawn at a position corresponding to the AR marker 85 in the captured image I.
[0164] Moreover, the display control unit 54 includes a 3D model drawing position control unit 54e instead of the 3D model drawing direction control unit 54b.
[0165] When overlap determination unit 53f predicts that subject 92 and 3D model 90M overlap, 3D model drawing position control unit 54e draws 3D model 90M at a position where it does not overlap with subject 92.
[0166] [6-2. Processing flow of mobile devices] 19 is a flowchart showing an example of the flow of processing performed by a mobile terminal according to the sixth embodiment. The flow of processing will be explained step by step below.
[0167] The process flow from step S90 to step S92 in FIG. 19 is the same as the process flow from step S10 to step S12 in FIG. 9 described in the first embodiment, and therefore description thereof will be omitted.
[0168] If it is determined in step S92 that the AR marker 85 has been detected (step S92: Yes), the subject detection unit 53a detects the subject 92 from within the captured image I (step S93).
[0169] Next, the overlap determining unit 53f detects the positional relationship between the AR marker 85 and the subject 92 (step S94).
[0170] Furthermore, the overlap determination unit 53f determines whether the subject 92 appears in the captured image I and whether the AR marker 85 and the subject 92 overlap (step S95). If it is determined that the condition is satisfied (step S95: Yes), the process proceeds to step S97. On the other hand, if it is not determined that the condition is satisfied (step S95: No), the overlap determination unit 53f predicts that when the 3D model 90M is drawn at a position corresponding to the AR marker 85, the subject 92 and the 3D model 90M will overlap, and performs the process of step S96.
[0171] If the result of step S95 is No, the 3D model drawing position control unit 54e changes the drawing position of the 3D model 90M, which is originally drawn at a position corresponding to the AR marker 85, to a position that does not overlap with the subject 92 (for example, a position away from the subject 92) (step S96).
[0172] Then, the 3D model drawing position control unit 54e draws the selected 3D model 90M at the position changed in step S96. If the result of the determination in step S95 is Yes, the 3D model drawing position control unit 54e draws the selected 3D model 90M at a position corresponding to the AR marker 85 (step S97).
[0173] Then, the imaging / recording processing unit 51b takes a selfie shot in a state in which the subject 92 and the 3D model 90M do not overlap each other (step S98).
[0174] [6-3. Effects of the sixth embodiment] As described above, according to the mobile terminal 80 of the sixth embodiment, the subject feature detection unit 53 (detection unit) detects the position of the subject 92 in the captured image I, which is a feature of the subject 92. Then, the display control unit 54 (control unit) changes the display position of the 3D model 90M to a position that does not overlap with the subject 92 when it is predicted that the position of the subject 92 and the position of the 3D model 90M overlap.
[0175] This allows selfie photography to be performed without overlapping subject 92 and 3D model 90M.
[0176] (7. Seventh embodiment) The seventh embodiment of the present disclosure is an example of a mobile terminal 80 (information processing device) having a function of correcting the size and scale of a 3D model 90M according to the size of a subject 92 in a captured image I, and taking a selfie in a state in which the subject 92 and the 3D model 90M appear to be the same height, or in a state in which the subject 92 appears to have a smaller face than the 3D model 90M. That is, the mobile terminal 80 of this embodiment changes the display mode (size and scale) of the 3D model 90M according to the size, which is a characteristic of the subject 92.
[0177] The hardware configuration of the mobile terminal 80 of this embodiment is the same as that of the mobile terminal 80 described in the first embodiment, and therefore a description thereof will be omitted. Note that, in order to realize the unique functions of this embodiment, the control program stored in the storage unit 38 uses a program different from the control program P1 (see FIG. 5).
[0178] [7-1. Functional configuration of mobile terminal] Fig. 20 is a functional block diagram showing an example of the functional configuration of a mobile terminal according to the sixth embodiment. A mobile terminal 80 according to the seventh embodiment has a functional configuration in which some functions are exchanged with those of the mobile terminal 80 according to the first embodiment (see Fig. 6). Only the differences in the functional configuration from the mobile terminal 80 according to the first embodiment will be described below.
[0179] The subject feature detection section 53 includes a positional relationship detection section 53g instead of the gaze direction detection section 53b.
[0180] The positional relationship detection unit 53g detects the positional relationship between the position of the AR marker 85 detected by the AR marker detection unit 52 and the position of the subject 92 detected by the subject detection unit 53a. The positional relationship detection unit 53g also detects the face size of the subject 92 occupying the angle of view of the camera 84.
[0181] Moreover, the display control unit 54 includes a 3D model drawing position control unit 54e instead of the 3D model drawing direction control unit 54b.
[0182] The 3D model drawing position control unit 54e sets the scale and drawing position of the 3D model 90M when drawing the 3D model 90M in the captured image I. Specifically, the drawing position of the 3D model 90M is set to match the height of the subject 92. In addition, the drawing position of the 3D model 90M is set so that the face position of the subject 92 and the face position of the 3D model 90M are at the same height. Of course, the drawing position may be set based on conditions other than these.
[0183] [7-2. Processing flow of mobile devices] 21 is a flowchart showing an example of the flow of processing performed by a mobile terminal according to the seventh embodiment. The flow of processing will be explained step by step below.
[0184] The process flow from step S100 to step S102 in FIG. 21 is the same as the process flow from step S10 to step S12 in FIG. 9 described in the first embodiment, and therefore will not be described.
[0185] If it is determined in step S102 that the AR marker 85 has been detected (step S102: Yes), the subject detection unit 53a detects the subject 92 from within the captured image I (step S103).
[0186] Next, the positional relationship detection unit 53g detects the positional relationship between the AR marker 85 and the subject 92 (step S104).
[0187] Furthermore, positional relationship detection unit 53g detects the face size of subject 92 occupying the angle of view of camera 84 (step S105).
[0188] Then, the positional relationship detection unit 53g judges whether the captured image I is a bust-up photograph showing the subject 92 from the chest up (step S106). If it is judged to be a bust-up photograph (step S106: Yes), the process proceeds to step S107. On the other hand, if it is not judged to be a bust-up photograph (step S106: No), the process proceeds to step S108.
[0189] If the result of the determination in step S106 is Yes, the 3D model drawing position control unit 54e sets the scale and drawing position when drawing the 3D model 90M so that the position of the face of the 3D model video is at the same height as the face of the subject 92 (step S107). Then, the process proceeds to step S109.
[0190] On the other hand, if the result of the determination in step S106 is No, the 3D model rendering position control unit 54e sets the scale and rendering position of the 3D model video so as to match the height of the subject 92 (step S108).
[0191] Then, the 3D model rendering position control unit 54e renders the selected 3D model 90M in the captured image I under the conditions set in step S107 or step S108 (step S109).
[0192] Then, the imaging / recording processing unit 51b captures a selfie with the subject 92 and the 3D model 90M aligned in a well-balanced manner (step S110).
[0193] [7-3. Effects of the Seventh Embodiment] As described above, according to the mobile terminal 80 of the seventh embodiment, the subject feature detection unit 53 (detection unit) detects the position and size of the subject 92 in the captured image I, which are features of the subject 92. Then, the display control unit 54 (control unit) adjusts the display position and size of the 3D model 90M in accordance with the position and size of the subject 92.
[0194] This allows selfie shooting to be performed with the subject 92 and the 3D model 90M aligned in a well-balanced manner.
[0195] Each of the above-described embodiments may have the functions of a plurality of different embodiments in combination, in which case the mobile terminal 80 has all the functional configurations of the plurality of embodiments.
[0196] 8. Eighth embodiment The eighth embodiment of the present disclosure is an example of an information processing device 81a having a function of forming a virtual space in a captured image captured by a camera 84 and displaying an avatar of a subject 92 and a 3D model 90M in the virtual space. Note that the avatar of the subject 92 displayed in the captured image is movable in the virtual space based on an operation of the subject 92.
[0197] [8-1. Overview of information processing device] FIG. 22 is a diagram illustrating an overview of an information processing apparatus according to the eighth embodiment.
[0198] The camera 84 connected to the information processing device 81a captures an image of the user, who is the subject 92, and generates a captured image Ia (virtual image). The generated captured image Ia is wirelessly transmitted and displayed on a head mounted display 89 (HMD) worn by the subject 92. The captured image Ia displayed on the head mounted display 89 displays an avatar 92a (first avatar) that is an alter ego of the subject 92, and also displays the 3D model 90M described in each embodiment of the present disclosure superimposed thereon. Note that the avatar 92a is a full CG video. The head mounted display 89 also has a function of detecting the movement of the head and the line of sight of the subject 92, and also has an audio output function of transmitting audio information output by the information processing device 81a to the subject 92.
[0199] The subject 92 also wears a controller 88 equipped with an acceleration sensor for detecting the subject's position, posture, movement, etc., and an operation switch for issuing an operation instruction to the information processing device. The line of sight direction and head movement of the subject 92 detected by the head mounted display 89, and the movement and operation information of the subject 92 detected by the controller 88 are wirelessly transmitted to the information processing device 81a and are reflected in the movement of the avatar 92a and the movement of the information processing device 81a. That is, the user who is the subject 92 can freely move around in the virtual space formed in the captured image Ia while viewing the captured image Ia. That is, the movement of the subject 92 is reflected in the movement of the avatar 92a.
[0200] Particularly, in the information processing device 81a of the present embodiment, the avatar 92a has a function of capturing an image in a virtual space using the virtual camera 84V. That is, when the subject 92 issues an instruction to operate the virtual camera 84V via the controller 88, the avatar 92a takes out the virtual camera 84V and captures an image in the direction indicated by the subject 92 in the virtual space.
[0201] When the line of sight direction E of the avatar 92a reflected in the captured image Ia captured by the virtual camera 84V faces the virtual camera 84V, the 3D model 90M is changed to a 3D model 90N whose line of sight direction F faces the virtual camera 84V. Then, the information processing device 81a captures a selfie of the state in which the avatar 92a and the 3D model 90N are both looking at the virtual camera 84V.
[0202] The hardware configuration of the information processing device 81a is the same as that of the mobile terminal 80 described in the first embodiment (FIG. 5), except that the liquid crystal display 16 is replaced with a head-mounted display 89 and the touch panel 14 is replaced with a controller 88 (not shown). The CPU 32 of the information processing device 81a is connected to the head-mounted display 89 and the controller 88 by wireless communication or wired communication.
[0203] [8-2. Functional configuration of information processing device] 23 is a functional block diagram showing an example of a functional configuration of an information processing device according to an eighth embodiment. The information processing device 81a according to the eighth embodiment includes an imaging unit 51, a subject feature detection unit 53, and a display control unit 54. The information processing device 81a further performs a process of generating an avatar 92a of a subject 92 using full CG, but the process of generating the avatar 92a may be performed according to a known method, and therefore a description thereof will be omitted.
[0204] The imaging unit 51 monitors video, captures images, and records video using a virtual camera 84V operated by an avatar 92a of the subject 92 in accordance with an operational instruction from the subject 92. The imaging unit 51 further includes a virtual video generation unit 51c, a virtual video monitor unit 51d, a virtual camera video generation unit 51e, and an imaging / recording processing unit 51b.
[0205] The virtual video generating unit 51c generates a captured image Ia (virtual image) captured by the virtual camera 84V.
[0206] The virtual video monitor unit 51d displays the captured image Ia on the head mounted display 89 in real time, ie, performs a so-called live view.
[0207] The imaging / recording processing section 51b stores the captured image Ia captured by the virtual camera 84V. The imaging / recording processing section 51b also performs selfie photography.
[0208] The subject feature detection unit 53 detects an avatar 92a of the subject 92 from a captured image Ia captured by the virtual camera 84V. The subject feature detection unit 53 further includes an avatar detection unit 53h and a gaze direction detection unit 53b.
[0209] The avatar detection unit 53h detects the avatar 92a from within the captured image Ia captured by the virtual camera 84V.
[0210] The gaze direction detection unit 53b detects the gaze direction E of the avatar 92a detected by the avatar detection unit 53h.
[0211] The display control unit 54 changes the display mode of the 3D model 90M according to the line of sight direction E of the avatar 92a detected by the subject feature detection unit 53. Specifically, when the line of sight direction E of the avatar 92a faces the virtual camera 84V, that is, when the avatar 92a is looking at the virtual camera, the display control unit 54 changes the line of sight direction F of the 3D model 90M to a direction facing the virtual camera 84V. The display control unit 54 further includes a 3D model frame selection unit 54a and a 3D model drawing direction control unit 54b. The display control unit 54 is an example of a control unit in the present disclosure.
[0212] The 3D model frame selection unit 54a selects a predetermined 3D model 90M from among a plurality of 3D models M stored in the storage unit 38. The 3D model frame selection unit 54a also selects a 3D model 90M having a line of sight direction F corresponding to the line of sight direction E of the avatar 92a detected by the line of sight direction detection unit 53b. More specifically, when the avatar 92a is looking at the virtual camera, the 3D model frame selection unit 54a selects a 3D model 90M in which the line of sight direction F of the 3D model 90M is oriented toward the virtual camera 84V.
[0213] The 3D model drawing direction control unit 54b superimposes the 3D model 90M selected by the 3D model frame selection unit 54a on the captured image Ia and displays it. The 3D model drawing direction control unit 54b also rotates the 3D model 90M to change the orientation of the 3D model 90M and generate a 3D model 90N. The 3D model drawing direction control unit 54b also superimposes the 3D model 90N whose orientation has been changed on the captured image Ia and displays it.
[0214] [8-3. Processing flow performed by information processing device] 24 is a flowchart showing an example of the flow of processing performed by the information processing device according to the eighth embodiment. The flow of the processing will be explained step by step below.
[0215] Upon receiving an operation instruction from subject 92 via controller 88, virtual video monitor unit 51d activates virtual camera 84V (step S120).
[0216] The virtual video generating unit 51c generates a captured image Ia (virtual image) viewed from the virtual camera 84V (step S121).
[0217] The avatar detection unit 53h detects the avatar 92a from within the captured image Ia. Then, the gaze direction detection unit 53b detects the gaze direction E of the detected avatar 92a (step S122).
[0218] Next, the line of sight direction detection unit 53b determines whether the avatar 92a is looking at the virtual camera 84V, i.e., whether the avatar 92a is looking at the virtual camera (step S123). If it is determined that the avatar 92a is looking at the virtual camera (step S123: Yes), the process proceeds to step S125. On the other hand, if it is not determined that the avatar 92a is looking at the virtual camera (step S123: No), the process proceeds to step S124.
[0219] If the result of the determination in step S123 is No, the 3D model frame selection unit 54a selects a video frame of the 3D model 90M from the 3D models M stored in the storage unit 38. Then, the 3D model drawing direction control unit 54b draws the 3D model 90M selected by the 3D model frame selection unit 54a at a predetermined position of the captured image Ia and plays the video frames in a loop (step S124). After that, the process returns to step S122.
[0220] On the other hand, if the result of step S123 is Yes, the 3D model drawing direction control unit 54b draws the 3D model 90M at a predetermined position of the captured image Ia and orients the 3D model 90M in the direction of the virtual camera 84V (step S125).
[0221] Next, the 3D model drawing direction control unit 54b draws the 3D model 90N selected by the 3D model frame selection unit 54a, with its eyes open and its line of sight facing the virtual camera 84V, at the drawing position set in step S125 (step S126). That is, the captured image Ib (virtual image) shown in FIG. 22 is generated.
[0222] Then, the imaging / recording processing unit 51b takes a selfie photograph while both the avatar 92a and the 3D model 90N are looking at the virtual camera (step S127).
[0223] [8-4. Effects of the eighth embodiment] As described above, in the information processing device 81a of the eighth embodiment, the subject feature detection unit 53 (detection unit) detects the line of sight E (feature) of the avatar 92a (first avatar) that exists in the same virtual space as the 3D model 90M displayed in the captured image Ia captured by the virtual camera 84V (second imaging unit) existing in the virtual space and that moves in the virtual space based on the operation of the subject 92 (user). Then, the display control unit 54 (control unit) changes the display mode of the 3D model 90M according to the detected line of sight E of the avatar 92a.
[0224] This allows the 3D model 90M displayed in the captured image Ia, including the avatar 92a (first avatar) that is the alter ego of the subject 92, to be able to perform interactive movements and reactions.
[0225] 9. Ninth embodiment The ninth embodiment of the present disclosure is an example of an information processing device that has a function of superimposing and displaying an avatar (second avatar) of a subject different from subject 92, instead of superimposing and displaying a 3D model 90M on the captured image Ia (virtual image) in the eighth embodiment.
[0226] [9-1. Overview of information processing device] FIG. 25 is a diagram illustrating an overview of an information processing apparatus according to the ninth embodiment.
[0227] The information processing device 81b includes a camera 84b having the same function as the camera 84a, in addition to the camera 84a having the function described in the eighth embodiment. The camera 84b captures an image of a subject 93 different from the subject 92. Note that, although the subject 92 and the subject 93 are standing side by side in Fig. 25, the subject 92 and the subject 93 may be located apart from each other. That is, the camera 84a and the camera 84b may be located apart from each other and have a function of wirelessly transmitting the captured image to the information processing device 81b.
[0228] The subject 93, like the subject 92, wears a head mounted display 89. The captured image Ia generated by the information processing device 81b is displayed in real time on the head mounted display 89. The head mounted display 89 worn by the subjects 92 and 93 has a function of outputting not only the captured image Ia but also audio information output by the information processing device 81b. The subject 93 also wears the same controller 88 as the subject 92.
[0229] As described in the eighth embodiment, the information processing device 81b generates a captured image Ia (virtual image) in which an avatar 92a (first avatar) of the subject 92 is displayed. Furthermore, the information processing device 81b displays an avatar 93a (second avatar) of the subject 93 in the captured image Ia. That is, the subject 92 and the subject 93 exist in the same virtual space, the captured image Ia, as avatars 92a and 93a, respectively.
[0230] The movement of the subject 92 is detected by the controller 88 and the head mounted display 89 and transmitted to the information processing device 81b. The movement of the subject 92 is reflected in the captured image Ia as the movement of the avatar 92a. Similarly, the movement of the subject 93 is detected by the controller 88 and the head mounted display 89 and transmitted to the information processing device 81b. The movement of the subject 93 is reflected in the captured image Ia as the movement of the avatar 93a. In this way, the subjects 92 and 93 move freely around in the captured image Ia as the avatars 92a and 93a, respectively.
[0231] The information processing device 81b has a function in which the avatar 92a captures an image in a virtual space using a virtual camera 84V. That is, when the subject 92 issues an instruction to operate the virtual camera 84V via the controller 88, the avatar 92a takes out the virtual camera 84V and captures an image in the direction instructed by the subject 92 in the virtual space. The captured captured image Ia is transmitted by the information processing device 81b to the head mounted display 89 worn by the subjects 92 and 93. Then, both of the subjects 92 and 93 observe the captured image Ia in real time.
[0232] The information processing device 81b detects the line of sight direction of the avatar 92a and the line of sight direction of the avatar 93a reflected in the captured image Ia captured by the virtual camera 84V. Then, when the avatars 92a and 93a are both looking at the virtual camera 84V, the information processing device 81b, in accordance with an operation instruction from the subject 92, captures a selfie of a captured image Ib in which the avatars 92a and 93a are both looking at the virtual camera 84V.
[0233] The hardware configuration of the information processing device 81b is the same as that of the information processing device 81a described in the eighth embodiment, to which a camera 84b, a controller 88 for the subject 93, and a head-mounted display 89 have been added.
[0234] [9-2. Functional configuration of information processing device] 26 is a functional block diagram showing an example of a functional configuration of an information processing device according to the ninth embodiment. The information processing device 81b according to the ninth embodiment includes an imaging unit 51, a subject feature detection unit 53, and a display control unit .
[0235] The imaging unit 51 has the same functions as in the eighth embodiment, and therefore a description thereof will be omitted. The subject feature detection unit 53 has the same configuration as the subject feature detection unit 53 described in the eighth embodiment, including an avatar detection unit 53h and a gaze direction detection unit 53b. However, in this embodiment, the avatar detection unit 53h detects the avatar 92a and the avatar 93a from the captured image Ia captured by the virtual camera 84V. In addition, the gaze direction detection unit 53b detects the gaze direction E of the avatar 92a and the gaze direction F of the avatar 93a detected by the avatar detection unit 53h.
[0236] The display control unit 54 displays the avatar 92a and the avatar 93a in a superimposed manner on the captured image Ia.
[0237] [9-3. Processing flow performed by information processing device] 27 is a flowchart showing an example of the flow of processing performed by an information processing device according to the 9th embodiment. The flow of processing will be explained step by step below.
[0238] The processes in steps S130 and S131 are the same as those in steps S120 and S121 described in the eighth embodiment, and therefore will not be described.
[0239] Next, the avatar detection unit 53h detects the avatar 92a (first avatar) from within the captured image Ia. Then, the gaze direction detection unit 53b detects the gaze direction E of the detected avatar 92a (step S132).
[0240] Next, the line of sight direction detection unit 53b determines whether the avatar 92a is looking at the virtual camera 84V, i.e., whether the avatar 92a is looking at the virtual camera (step S133). If it is determined that the avatar 92a is looking at the virtual camera (step S133: Yes), the process proceeds to step S135. On the other hand, if it is not determined that the avatar 92a is looking at the virtual camera (step S133: No), the process proceeds to step S134.
[0241] If the result of the determination in step S133 is No, the display control unit 54 loop-plays (live view displays) the avatar 93a in the captured image Ia (step S134), and then returns to step S132.
[0242] On the other hand, if the result of the determination in step S133 is Yes, the avatar detection unit 53h detects the avatar 93a (second avatar) from the captured image Ia. Then, the gaze direction detection unit 53b detects the gaze direction F of the detected avatar 93a (step S135). Note that when it is determined that the avatar 92a is looking at the virtual camera, the information processing device 81b may give an instruction to the subject 93 via the head mounted display 89, such as "Let's take a picture together."
[0243] Next, the line of sight direction detection unit 53b determines whether the avatar 93a is looking at the virtual camera 84V, i.e., whether the avatar 93a is looking at the virtual camera (step S136). If it is determined that the avatar 93a is looking at the virtual camera (step S136: Yes), the process proceeds to step S137. On the other hand, if it is not determined that the avatar 93a is looking at the virtual camera (step S136: No), the process returns to step S135.
[0244] If the result of the determination in step S136 is Yes, the imaging / recording processing unit 51b takes a selfie photograph with both the avatar 92a and the avatar 93a looking at the virtual camera (step S137).
[0245] [9-4. Effects of the ninth embodiment] As described above, in the information processing device 81b of the ninth embodiment, the display control unit 54 (control unit) displays the avatar 93a (second avatar) of the subject 93 different from the subject 92 in the captured image Ia.
[0246] This allows a subject 92 (user) to perform interactive actions and reactions with another subject 93 in the virtual space.
[0247] In addition, the effects described in this specification are merely examples and are not intended to be limiting, and other effects may be achieved. In addition, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit and scope of the present disclosure.
[0248] For example, the present disclosure can also be configured as follows.
[0249] (1) A control unit that controls a 3D model displayed in a captured image; a detection unit that detects features of a subject displayed in the captured image together with the 3D model, The control unit is an information processing device that changes a display manner of the 3D model depending on the characteristics of the subject detected by the detection unit. (2) The detection unit detects a gaze direction, which is a characteristic of the subject, The control unit changes the orientation of the 3D model according to the detected line of sight of the subject. The information processing device according to (1). (3) The control unit changes the orientation of the 3D model by rotating the 3D model according to the line of sight of the subject. The information processing device according to (2). (4) The control unit changes the orientation of the 3D model by rotating a coordinate system in which the 3D model is placed, according to the line of sight of the subject. The information processing device according to (2). (5) When the detection unit detects that the line of sight of the subject is directed in the direction in which the captured image was captured, The control unit changes the orientation of the 3D model to the direction in which the captured image was captured. The information processing device according to any one of (1) to (4). (6) When the detection unit detects that the gaze direction of the subject is directed toward the 3D model, The control unit changes the orientation of the 3D model to face the subject. The information processing device according to any one of (1) to (4). (7) The detection unit detects a facial expression that is a characteristic of the subject, When the control unit detects that the facial expression of the subject is a smile, the control unit changes the orientation of the 3D model to the direction in which the captured image was captured. The information processing device according to any one of (1) to (6). (8) The detection unit detects brightness, which is a characteristic of the subject, The control unit changes the brightness of the 3D model according to the brightness of the subject. The information processing device according to any one of (1) to (7). (9) The detection unit detects a distance to a subject, which is a feature of the subject, The control unit changes a distance to the 3D model so that the 3D model and the subject are within a depth of field of the captured image. The information processing device according to any one of (1) to (8). (10) The detection unit detects a position of the subject in the captured image, which is a feature of the subject, When it is predicted that a position of the subject and a position of the 3D model overlap, the control unit changes a display position of the 3D model to a position that does not overlap with the subject. The information processing device according to any one of (1) to (9). (11) The detection unit detects a position and a size of the subject in the captured image, which are characteristics of the subject, The control unit adjusts a display position and a size of the 3D model according to a position and a size of the subject. The information processing device according to any one of (1) to (10). (12) The control unit further includes an imaging unit that captures an image of the 3D model and the subject after the control unit changes the display mode of the 3D model. The information processing device according to any one of (1) to (11). (13) The imaging unit is a camera existing in the real world or a virtual camera existing in the virtual world. The information processing device according to (12). (14) the subject is a first avatar that exists in the same virtual space as the 3D model and moves in the virtual space based on a user's operation, The detection unit detects a feature of the first avatar displayed in an image captured by a second imaging unit present in the virtual space. The information processing device according to any one of (1) to (13). (15) The 3D model is a second avatar that operates in the virtual space based on the operation of a subject different from the subject. The information processing device according to (14). (16) The 3D model is a model having 3D information of an object existing in the real world, generated using a plurality of viewpoint images captured from different viewpoints of the object. The information processing device according to any one of (1) to (14). (17) The control unit is When a predetermined display is displayed in the captured image, the 3D model is displayed superimposed on the captured image. The information processing device according to any one of (1) to (13). (18) a control process for controlling a 3D model displayed in the captured image; a detection process for detecting features of a subject displayed in the captured image together with the 3D model; The control process is an information processing method in which the display manner of the 3D model is changed depending on the characteristics of the subject detected by the detection process. (19) Computer, A control unit that controls a 3D model displayed in a captured image; A detection unit that detects features of a subject displayed in the captured image together with the 3D model, A program that causes the control unit to change the display mode of the 3D model according to the characteristics of the subject detected by the detection unit. [Explanation of symbols]
[0250] 51...imaging unit, 52...AR marker detection unit, 53...subject feature detection unit (detection unit), 54...display control unit (control unit), 80...mobile terminal (information processing device), 81a, 81b...information processing device, 82...display screen, 84, 84a, 84b...camera (first imaging unit), 84V...virtual camera (second imaging unit), 85...AR marker (predetermined display), 90...subject (object), 90M, 90N, M...3D model, 92...subject (user), 92a...avatar (first avatar), 93...subject, 93a...avatar (second avatar), Ia, Ib...captured image (virtual image), E, F...gaze direction, I...captured image
Claims
1. a display control unit that is disposed in a three-dimensional space and controls a 3D model displayed in a captured image; a detection unit that detects line-of-sight information of a subject displayed in the captured image together with the 3D model; an imaging unit that captures an image of the 3D model and the subject displayed in the captured image, When the gaze direction of the subject detected by the detection unit is directed toward the imaging unit, the display control unit changes the display mode of the 3D model so that the gaze direction of the 3D model approaches the direction being imaged, and then captures an image using the imaging unit; or, when the gaze direction of the subject detected by the detection unit is directed toward the 3D model, the display control unit changes the display mode of the 3D model so that the gaze direction of the 3D model approaches the direction of the subject, and then captures an image using the imaging unit.
2. The display control unit changes the orientation of the 3D model in response to line-of-sight information of the detected subject. The information processing device according to claim 1 .
3. The display control unit changes an orientation of the 3D model by rotating the 3D model in accordance with line-of-sight information of the subject. The information processing device according to claim 2 .
4. The display control unit changes the orientation of the 3D model by rotating a coordinate system in which the 3D model is placed, in accordance with line-of-sight information of the subject. The information processing device according to claim 2 .
5. When the detection unit detects that the line of sight information of the subject is directed in the direction in which the captured image was captured, The display control unit changes the orientation of the 3D model to a direction in which the captured image was captured. The information processing device according to claim 2 .
6. When the detection unit detects that the line of sight information of the subject is directed toward the 3D model, The display control unit changes the orientation of the 3D model to face the subject. The information processing device according to claim 2 .
7. The detection unit detects a facial expression that is a characteristic of the subject, When it is determined that the facial expression of the subject is a smile, the display control unit changes the orientation of the 3D model to a direction in which the captured image was captured. The information processing device according to claim 1 .
8. The detection unit detects brightness, which is a characteristic of the subject, The display control unit changes the brightness of the 3D model in accordance with the brightness of the subject. The information processing device according to claim 1 .
9. The detection unit detects a distance to a subject, which is a feature of the subject, The display control unit changes a distance to the 3D model so that the 3D model and the subject are within a depth of field of the captured image. The information processing device according to claim 1 .
10. The detection unit detects a position of the subject in the captured image, which is a feature of the subject, When it is predicted that a position of the subject and a position of the 3D model overlap, the display control unit changes a display position of the 3D model to a position that does not overlap with the subject. The information processing device according to claim 1 .
11. The detection unit detects a position and a size of the subject in the captured image, which are characteristics of the subject, The display control unit adjusts a display position and a size of the 3D model according to a position and a size of the subject. The information processing device according to claim 1 .
12. The imaging unit is a camera existing in the real world or a virtual camera existing in the virtual world. The information processing device according to claim 1 .
13. the subject is a first avatar that exists in the same virtual space as the 3D model and moves in the virtual space based on a user's operation, the detection unit detects a feature of the first avatar displayed in an image captured by a second imaging unit present in the virtual space; The information processing device according to claim 1 .
14. The 3D model is a second avatar that operates in the virtual space based on an operation of a subject different from the subject. The information processing device according to claim 13.
15. The 3D model is a model having 3D information of an object existing in the real world, generated using a plurality of viewpoint images captured from different viewpoints of the object. The information processing device according to claim 1 .
16. The display control unit is When a predetermined display is detected in the captured image, the 3D model is displayed superimposed on the captured image. The information processing device according to claim 15.
17. a display control process that controls a 3D model that is arranged in a three-dimensional space and displayed in a captured image; a detection process for detecting line-of-sight information of a subject displayed in the captured image together with the 3D model; an imaging process of imaging the 3D model and the subject displayed in the captured image by an imaging unit; The information processing method wherein, when the gaze direction of the subject detected by the detection process is directed toward the imaging unit, the display control process changes the display mode of the 3D model so that the gaze direction of the 3D model approaches the direction being imaged, and then captures an image using the imaging process; or, when the gaze direction of the subject detected by the detection process is directed toward the 3D model, the display control process changes the display mode of the 3D model so that the gaze direction of the 3D model approaches the direction of the subject, and then captures an image using the imaging process.
18. Computer, a display control unit that is disposed in a three-dimensional space and controls a 3D model displayed in a captured image; a detection unit that detects line-of-sight information of a subject displayed in the captured image together with the 3D model; An imaging unit that captures an image of the 3D model and the subject displayed in the captured image; It also functions as a When the gaze direction of the subject detected by the detection unit is directed toward the imaging unit, the display control unit changes the display mode of the 3D model so that the gaze direction of the 3D model approaches the direction being imaged, and then captures an image using the imaging unit; or, when the gaze direction of the subject detected by the detection unit is directed toward the 3D model, the display control unit changes the display mode of the 3D model so that the gaze direction of the 3D model approaches the direction of the subject, and then captures an image using the imaging unit.
Citation Information
Patent Citations
Image processor and image processing program
JP2006227838A
Image processing method and image processor
JP2007042055A
Image processor, stereoscopic image printing system, image processing method and program
JP2012160039A
Systems and methods for augmented and virtual reality
JP2015503141A
Video game processing program, video game processing system, and video game processing method
JP2017056114A
