Portable terminal, augmented reality object display control method, and program

Abstract

A portable terminal according to the present invention comprises a region size–specifying unit that is configured to extract a region image of a predetermined region of a user from a user image that is an image that has been captured of the user and to specify the size of the extracted region image as a region size, an action detection unit that is configured to detect that a self device is in a stationary state, and a display control unit that is configured to control the display mode of an augmented reality object on a display. The display control unit is configured to change the display mode of the augmented reality object to a predetermined mode in accordance with a change in the region size in the stationary state from an initial size that is a pre-registered initial value for the region size.

Description

Mobile terminal, augmented reality object display control method and program

[0001] [Description of Related Applications] The present invention is based on priority claim of Japanese Patent Application No. 2024-018310 (filed February 9, 2024), the entire contents of which are incorporated herein by reference. The present invention relates to a mobile terminal, an augmented reality object display control method, and a program.

[0002] On the display of a mobile device such as a smartphone, an augmented reality (AR) object may be superimposed on an image captured of a real space. For example, in a store, product information that cannot be displayed on a display shelf may be provided to a user's mobile device using augmented reality (AR).

[0003] When changing the size of an AR object displayed on the display of a mobile device, the user needs to perform a pinch operation or the like on the display. However, the user is not always in a situation where the operation on the display is possible. For example, when both hands are occupied, or when operating the mobile device with one hand and the fingers cannot reach the object, or when the hands are dry or wet, or when wearing gloves, the user cannot perform a pinch operation to change the size of the AR object, which makes the device inconvenient to use.

[0004] There is a technology that allows a user to interact with an AR object without performing operations such as touching a display (see, for example, Patent Document 1). The technology disclosed in Patent Document 1 "uses a radar system to accurately determine three-dimensional (3D) gestures that can be used to interact with an augmented reality (AR) object presented on the display of an electronic device such as a smartphone. The user can perform the 3D gestures from a distance, without having to hold the electronic device steady while touching the display, and the gestures do not obstruct the user's view of the AR object presented on the display (abstract excerpt)."

[0005] Japanese Patent Application Laid-Open No. 2022-119986

[0006] The following analysis has been carried out by the inventors of the present invention.

[0007] According to the technology disclosed in Patent Document 1, a user can give instructions to an AR object without having to give various operational instructions on a display. However, the technology disclosed in Patent Document 1 adds a radar system to a mobile device, which is used to grasp the user's operations on the AR object. This radar system is a special, large-scale system that outputs radar signals and detects their reflections. This technology cannot be implemented on general mobile devices that do not have such a system installed.

[0008] The present invention has been made in consideration of the above circumstances, and aims to provide a technology that improves the usability of mobile terminals with a simple configuration and regardless of the situation, without adding new hardware.

[0009] According to a first aspect of the present invention, there is provided a mobile terminal comprising: a part size determination unit configured to extract part images of predetermined parts of a user from a user image, which is an image of the user, and determine the size of the extracted part image as the part size; a motion detection unit configured to detect that the device is in a stationary state; and a display control unit configured to control the display manner of an augmented reality object on a display, wherein the display control unit is configured to change the display manner of the augmented reality object to a predetermined manner in accordance with a change in the part size from an initial size, which is a pre-registered initial value of the part size, during the stationary state.

[0010] According to a second aspect of the present invention, there is provided an augmented reality object display control method in a mobile terminal, which extracts a part image of a predetermined part of a user from an image of the user, and each time an image is extracted, specifies the size of the extracted part image as the part size, and while the mobile terminal is stationary, changes the display manner of the augmented reality object on the display of the mobile terminal in accordance with a change in the specified part size from an initial size, which is an initial value of the part size that has been registered in advance.

[0011] According to a third aspect of the present invention, there is provided a program for causing a computer mounted on a mobile terminal to function as: a part size determination unit that extracts part images of predetermined parts of a user from an image of the user and determines the size of the extracted part image as the part size; a motion detection unit that detects that the device is in a stationary state; and an augmented reality display control unit that controls the display manner of an augmented reality object on a display to change to a predetermined manner in accordance with a change in the part size from an initial size, which is a pre-registered initial value for the part size, during the stationary state.

[0012] The program can be recorded on a computer-readable storage medium. The storage medium can be a non-transitory medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. The present invention can also be embodied as a computer program product.

[0013] According to the present invention, it is possible to improve the usability of a mobile terminal with a simple configuration, regardless of the situation, without adding new hardware.

[0014] 1A is an explanatory diagram illustrating an example of an overview of the present disclosure, and FIG. 1B is a functional block diagram of an example of a mobile terminal of the present disclosure. FIG. 1C is a flowchart of an example of an AR display mode control process of the present disclosure. FIG. 1A and FIG. 1B are external views of an example of a housing of a smartphone of the present disclosure, and FIG. 1C is a functional block diagram of an example of a smartphone of the present disclosure. FIG. 1C is an explanatory diagram illustrating an example of an AR display mode control process of the present disclosure. FIG. 1C is a functional block diagram of an example of a smartphone of the present disclosure. FIG. 1C is a flowchart of an example of an AR display mode control process of the present disclosure. FIG. 1C is an explanatory diagram illustrating a specific example of the present disclosure. FIG. 1C is an explanatory diagram illustrating another specific example of the present disclosure. FIG. 1C is an explanatory diagram illustrating an example of a table of a modified example of the present disclosure. FIG. 1C is a configuration diagram illustrating an example of a hardware configuration of a smartphone of the present disclosure.

[0015] An overview of one embodiment of the present invention (hereinafter referred to as the present embodiment) will be described below with reference to the drawings. Note that reference numerals in the drawings are assigned to each element for convenience as an example to facilitate understanding, and are not intended to limit the present invention to the illustrated form. Furthermore, connection lines between blocks in the drawings and the like referred to in the following description include both bidirectional and unidirectional lines. Unidirectional arrows are used to schematically indicate the flow of the main signal (data) and do not exclude bidirectionality.

[0016] In addition, although there are ports and interfaces at the connection points of input and output of each block in the figure, they are not shown. In the following explanation, "A and / or B" means A or B, or A and B.

[0017] First Embodiment First, a first embodiment of the present invention will be described.

[0018] First, an overview of the present embodiment will be described. When a user 300 moves a predetermined part of the mobile terminal 100a while the mobile terminal 100a is fixed, the mobile terminal 100a changes the display mode of an augmented reality (AR) object 410 displayed on the mobile terminal 100a accordingly.

[0019] For example, a case where the mobile terminal 100a is a smartphone and the predetermined part is the face (head) is shown in Fig. 1(a). As shown in this figure, when the user 300 moves their face closer to the smartphone display while holding the smartphone (mobile terminal 100a) in a fixed position, the mobile terminal 100a increases the display size of the AR object 410 accordingly. On the other hand, when the user 300 moves their face away from the smartphone display, the mobile terminal 100a decreases the display size of the AR object 410.

[0020] The mobile terminal 100a according to the present embodiment that realizes this function will be described below. These functions are realized, for example, by installing a program (AR display control application (AR app)) that controls the display mode of the AR object 410 in the mobile terminal 100a.

[0021] FIG. 1B is a functional block diagram of functions that are realized by an AR application program when the AR application program is installed on the mobile terminal 100a of this embodiment.

[0022] As shown in the figure, the mobile terminal 100a of this embodiment includes a body part size specifying section 120a, a motion detecting section 130a, and a display control section 140a.

[0023] Body part size specifying section 120a extracts body part images of predetermined body parts of user 300 from a user image, which is an image of user 300. Then, it specifies the size of the extracted body part image as the body part size.

[0024] As described above, if the part is the face (head) of user 300, part size specification unit 120a first specifies the area of ​​user 300's face (face image) from the user image and extracts the specified face image. For example, existing face image detection technology is used to specify and extract the face image.

[0025] For example, the number of pixels is used as the part size. In the above example, the part size specifying unit 120a calculates the number of pixels in the face area of ​​the user image as the part size.

[0026] The user image is captured by, for example, a camera provided in the mobile terminal 100a. For example, if the mobile terminal 100a is a smartphone, the user image is captured by an internal camera.

[0027] The motion detector 130a detects that the mobile terminal 100a is stationary, i.e., that the mobile terminal 100a is fixed. This detection is performed, for example, by determining the output of a sensor provided in the mobile terminal 100a.

[0028] For example, if the mobile terminal 100a is a smartphone, the output (sensor signal) from a motion detection sensor provided in the smartphone, such as an acceleration sensor or a gyro sensor, is used. For example, if it is determined that the value of the sensor signal from the motion detection sensor has been below a certain threshold for a predetermined period of time, the motion detection unit 130a determines that the mobile terminal 100a is in a stationary state. In other words, it detects that the mobile terminal 100a is in a stationary state.

[0029] The display control unit 140a controls the display mode of the AR object 410 on the display of the mobile terminal 100a. In this embodiment, the display control unit 140a changes the display mode of the AR object 410 on the display in accordance with a change in the body part size from the initial size while the mobile terminal 100a is stationary.

[0030] The initial size is an initial value of the part size. For example, after the AR app is installed, the part size specified in the first acquired user image is registered as the initial size in the initial size storage unit 190a. The initial size may be specified from a user image acquired at a predetermined timing. The initial size storage unit 190a is provided, for example, in a storage device of the mobile terminal 100a.

[0031] For example, when the part size becomes larger than the initial size, the display control unit 140a increases the display size of the AR object 410. Also, when the part size becomes smaller than the initial size, the display control unit 140a decreases the display size of the AR object 410. At this time, the size of the captured image of real space itself, which is superimposed and displayed with the AR object 410, is not changed.

[0032] [AR Display Mode Control Processing] The flow of the AR display mode control processing by the display control unit 140a is shown in Fig. 2. This processing is started when the body part size identification unit 120a acquires a user image.

[0033] The part size specifying unit 120a detects a predetermined part (part image) from the acquired user image, and specifies the number of pixels thereof as the part size (step S1101).

[0034] The motion detector 130a determines whether the mobile terminal 100a is in a stationary state (step S1102). If the mobile terminal 100a is not in a stationary state (S1102; No), the process ends.

[0035] If the body part is stationary (S1102; Yes), the display control unit 140a determines whether the body part size identified in step S1101 has changed from the initial size (step S1103). If the body part size has not changed (S1103; No), the display control unit 140a ends the process.

[0036] If there has been a change (S1103; Yes), the display control unit 140a changes the display mode of the AR object 410 to a predetermined mode according to the mode of the change (step S1104), superimposes it on the captured image, and terminates the processing.

[0037] As described above, according to this embodiment, while the mobile terminal 100a is fixed, the user's movement is detected by image processing and the display mode of the AR object 410 is changed. Specifically, the mobile terminal 100a processes a user image obtained by capturing a specific part of the user 300 within the mobile terminal 100a to detect a change in the size of the specific part of the user 300 and recognizes this as an operation instruction. Then, the mobile terminal 100a changes the display mode of only the AR object 410 in response to the operation instruction.

[0038] Therefore, even if the user 300 cannot perform operations on the display screen of the mobile terminal 100a, the mobile terminal 100a can easily change the display mode of only the AR object 410.

[0039] Furthermore, according to this embodiment, when the mobile terminal 100a is in a stationary state and the size of a predetermined part of the user 300 changes in the user image, the mobile terminal 100a recognizes this as an operation instruction for the AR object 410. For example, when the user 300 wants to change the size of an object in a captured image of real space, the user 300 moves the mobile terminal 100a closer to or farther away from the object. In this case, too, the change in the relative distance between the mobile terminal 100a and the user 300 changes the size of the predetermined part of the user 300 in the user image.

[0040] However, according to this embodiment, the mobile terminal 100a recognizes a change in the size of a predetermined part in the user image as an operation instruction only when the mobile terminal 100a is stationary. That is, it is possible to clearly distinguish whether the change in size of the predetermined part is due to the movement of the mobile terminal 100a or whether the change in size of the predetermined part is due to the user 300 intentionally moving the predetermined part. Therefore, the mobile terminal 100a can accurately grasp the intention of the user 300 through simple processing.

[0041] Therefore, according to the present embodiment, the mobile terminal 100a can display the AR object 410 in a desired display mode on the screen of the mobile terminal 100a through simple processing without adding new hardware. Therefore, the usability of the mobile terminal 100a is improved with a simple configuration regardless of the situation.

[0042] <<Second Embodiment>> Next, a second embodiment will be described. This embodiment is an embodiment that embodies the first embodiment in more detail. In this embodiment, configurations with the same names as those in the first embodiment basically have the same functions as those in the first embodiment. Below, this embodiment will be described, focusing on the differences from the first embodiment.

[0043] Hereinafter, the present embodiment will be described using an example in which the mobile terminal is a smartphone 100. In addition, the specific part will be described using an example in which the face of the user 300 is set as the specific part, and the display mode to be changed is the size of the AR object 410.

[0044] 3( a) and 3(b) are external views of the housing of the smartphone 100 of this embodiment. As shown in Fig. 3(a), the smartphone 100 has an in-camera 211 and a display 212 on a first surface 210 side of the housing. As shown in Fig. 3(b), the smartphone 100 has an out-camera 221 on a second surface 220 side of the housing. The housing also includes an information processing device, a wireless communication device, an acceleration sensor, a gyro sensor, a GPS receiver, and the like, all of which are not shown.

[0045] 3C is a functional block diagram of functions related to this embodiment of the smartphone 100 of this embodiment. As described above, each function shown in this diagram is realized by installing an AR application.

[0046] As shown in the figure, the smartphone 100 includes an initial size registration unit 110, a body part size identification unit 120, a motion detection unit 130, a display control unit 140, a user image acquisition unit 150, and a captured image acquisition unit 160. Components with the same names as those in the first embodiment basically have the same functions as those in the first embodiment.

[0047] The user image acquisition unit 150 acquires an image (user image) captured by the in-camera 211. In this embodiment, for example, when an AR application is launched, the user image acquisition unit 150 first activates the in-camera 211. Thereafter, the in-camera 211 acquires images captured at predetermined time intervals, for example, and transmits the images to the body part size identification unit 120 as user images.

[0048] The captured image acquisition unit 160 activates the in-camera 211 or the out-camera 221 according to the processing content of the AR application, and acquires an image captured by either camera (a captured image). The captured image is an image of a real space, and is displayed on the display 212 with the AR object 410 superimposed thereon.

[0049] The captured image acquisition unit 160 acquires captured images at predetermined time intervals, for example, and transmits them to the display control unit 140 .

[0050] As in the first embodiment, the part size specification unit 120 extracts part images of predetermined parts of the user 300 from the user image. In this embodiment, a face image is extracted. Then, the size of the extracted face image is specified as the part size.

[0051] In this embodiment, the part size identification unit 120 processes the user image sent from the user image acquisition unit 150 at predetermined time intervals and extracts a face region as a face image. Each time an extraction is performed, the number of pixels in the face image is calculated to determine the part size. As in the first embodiment, various existing face detection methods can be used to detect the face region (face image).

[0052] If the transmitted user image includes, for example, multiple face images, the body part size specification section 120 may determine the largest face image as the face of the user 300 .

[0053] The body part size specifying section 120 outputs the specified body part size to the display control section 140 and the initial size registration section 110 .

[0054] The initial size registration unit 110 registers the part size identified by the part size identification unit 120 at a predetermined timing as the initial size in the initial size storage unit 190. In this embodiment, for example, after the AR app is started or the in-camera 211 is started, the part size output first from the part size identification unit 120 is registered as the initial size.

[0055] The motion detection unit 130 detects whether the smartphone 100 is in a stationary state. In this embodiment, the motion detection unit 130 determines whether the smartphone 100 is in a stationary state by using outputs from various sensors, such as an acceleration sensor and a gyro sensor, that the smartphone 100 is equipped with.

[0056] The motion detection unit 130 monitors the outputs of these sensors, for example, at predetermined time intervals. Then, for example, when an output indicating that the housing of the smartphone 100 is in a stationary state is detected continuously for a predetermined period, the motion detection unit 130 determines that the housing of the smartphone 100 is in a stationary state. Various existing methods can be used to determine the stationary state.

[0057] When the motion detection unit 130 determines that the smartphone 100 is in a stationary state, it outputs a still-state signal to the display control unit 140. In this embodiment, for example, it determines whether the smartphone 100 is in a stationary state at predetermined time intervals, and outputs a still-state signal to the display control unit 140 each time it determines that the smartphone 100 is in a stationary state. Therefore, for example, when the smartphone 100 continues to be in a stationary state, the motion detection unit 130 continues to output a still-state signal to the display control unit 140.

[0058] The display control unit 140 controls the display mode (display size) of the AR object 410 on the display 212 of the smartphone 100. In the present embodiment, as in the first embodiment, the display control unit 140 changes the display mode (display size) of the AR object 410 in accordance with a change in the part size (face image size) from the initial size while in a stationary state.

[0059] To achieve this, the display control unit 140 of this embodiment includes a part size change rate calculation unit 141 , a display mode determination unit 142 , and a display processing unit 143 .

[0060] The part size change rate calculation unit 141 calculates the rate of change (part size change rate) of the part size transmitted from the part size identification unit 120 from the initial size. In this embodiment, each time a part size is transmitted from the part size identification unit 120, the part size change rate is calculated using the initial size registered in the initial size storage unit 190. The calculated part size change rate is then transmitted to the display mode determination unit 142.

[0061] As an example, if the value of the region size to be transmitted is B and the value of the initial size is A, the region size change rate Rb is calculated as Rb = (B - A) / A x 100. Alternatively, it may be calculated as Rb = B / A.

[0062] The display mode determination unit 142 determines the display mode of the AR object 410 to be displayed on the display 212 in accordance with the part size change rate Rb calculated by the part size change rate calculation unit 141. Specifically, it calculates the change rate (display change rate) from the initial display mode of the AR object 410. The display change rate is, for example, the rate at which various display parameters specifying the initial display mode are changed from default values ​​(initial values).

[0063] In this embodiment, the display size of the AR object 410 is changed. Therefore, here, the display mode determination unit 142 calculates the enlargement ratio (or reduction ratio) with respect to the original display size (initial display size) as the display change ratio, and determines that ratio as the display change ratio.

[0064] The display change rate Rd is calculated, for example, as (region size change rate (Rb) / 100+1). For example, if the region size change rate Rb is 0%, the display change rate Rd is 1. If the region size change rate Rb is C% (C>0), the display change rate Rd is (C / 100+1). If the region size change rate Rb is -D%, the display change rate Rd is ((-D) / 100+1).

[0065] The display mode determination unit 142 outputs the determined display change rate Rd of the AR object 410 to the display processing unit 143 .

[0066] The display processing unit 143 changes the display mode of the AR object 410 using the display change rate Rd output from the display mode determination unit 142, generates a display screen, and outputs it to the display 212. For example, the display mode is changed by multiplying the display parameter by the display change rate Rd. In this embodiment, for example, the AR object 410 after the change in display mode is displayed superimposed on an image of the surroundings (captured image) acquired by the outer camera 221.

[0067] In this embodiment, the display parameter to be changed is size. For example, if the original size (default value of the display parameter) is L and the display change rate Rd is (C / 100+1), the display processing unit 143 enlarges the size of the AR object 410 by (C / 100+1) times to ((C / 100+1)×L) and generates a display screen. Also, if the display change rate Rd is ((-D) / 100+1), the display processing unit 143 reduces the size by ((-D) / 100+1) times to (((-D) / 100+1)×L) and generates a display screen.

[0068] At this time, the display processing unit 143 may adjust the display position of the AR object 410, etc. For example, the AR object 410 is basically displayed superimposed at a position (initial position) determined by default values ​​of the display parameters. However, if the size of the AR object 410 is enlarged according to the display change rate Rd and the AR object 410 is to be displayed at the initial position, there may be cases where the entire AR object 410 cannot be displayed within the display 212. In such cases, the display position of the AR object 410 may be displaced. Alternatively, the AR object 410 may be displayed in a scrollable manner.

[0069] The AR object 410 is held in advance in, for example, the AR object storage unit 180. Specifically, for example, each display parameter is stored in association with identification information that identifies the AR object 410.

[0070] In this embodiment, the part size change rate calculation unit 141 performs the above process only while a still signal is received from the motion detection unit 130. For example, while a still signal is not received from the motion detection unit 130, the part size change rate calculation unit 141 may output 0% as the part size change rate Rb. As a result, during this time, the AR object 410 is displayed with the default display parameters.

[0071] The above processing will be briefly explained. Fig. 4(a) is a diagram illustrating a change in the face image in the user image 310, and Fig. 4(b) is a diagram illustrating the accompanying change in the size of the displayed AR object 410. In each case, the left side is the initial value.

[0072] 4A, when the face image changes from face image 311 to face image 312, the part size change rate Rb and the display change rate Rd are calculated in the display control unit 140. Then, in accordance with the calculated display change rate Rd, the AR object 410 is enlarged from the default size 411 to a size 412 and displayed.

[0073] [AR Display Mode Control Process] Next, the flow of the AR display mode control process of this embodiment will be described. Here, the AR display mode control process is started when an AR application launch instruction is received. Fig. 5 shows the processing flow of the AR display mode control process of this embodiment.

[0074] In the following description, the predetermined part of the user 300 is assumed to be the face (head). The display size of the AR object 410 is assumed to change in accordance with a change in the size of the face image. The AR object 410 is assumed to be displayed superimposed on an image (captured image) acquired by the outer camera 221.

[0075] First, upon receiving an instruction from the user 300, the control unit (not shown) of the smartphone 100 starts the AR application (step S2101). For example, when both hands are occupied, the instruction to start the AR application may be given by a voice command or the like.

[0076] When the AR application is launched, the user image acquisition unit 150 and the captured image acquisition unit 160 respectively activate the in-camera 211 and the out-camera 221 (step S2102) and start image acquisition. Then, a counter n (n is an integer equal to or greater than 1) that counts the number of acquired images is initialized (n=1) (step S2103).

[0077] When the user image acquisition unit 150 acquires the user image 310 (step S2104), the part size determination unit 120 extracts a predetermined part as a part image and determines the size of the part image as the part size (step S2105). In this embodiment, as described above, a face image is extracted and the size (number of pixels) of the face image is determined.

[0078] If the extracted and identified face image is the initially acquired user image 310, i.e., if the counter n is 1 (step S2106; Yes), the initial size registration unit 110 registers the calculated size of the face image as the initial size in the initial size storage unit 190 (step S2107). Then, the user image acquisition unit 150 increments the value of n by 1 (step S2108) and returns to step S2104.

[0079] On the other hand, if n is other than 1 (S2106; No), the display control unit 140 determines whether the smartphone 100 is in a stationary state (step S2111). Here, it determines whether a stationary signal is received from the motion detection unit 130. Note that if a stationary signal is not received (S2111; No), the process proceeds to step S2115, which will be described later.

[0080] If it has been received (S2111; Yes), part size change rate calculation section 141 calculates part size change rate Rb (step S2112). In this embodiment, the calculation is performed using the part size of the face image received from part size identification section 120 and the initial size of the face image stored in initial size storage section 190 using the above method.

[0081] Next, the display mode determination unit 142 calculates the display change rate Rd of the display parameters using the part size change rate Rb (step S2113).

[0082] The display processing unit 143 then generates a display screen (step S2114). Here, the AR object 410 is superimposed on the latest captured image acquired by the captured image acquisition unit 160 with display parameters changed using the display change rate Rd.

[0083] The display control unit 140 returns to step S2108 and continues the process until an instruction to end the AR app is received (step S2115; Yes).

[0084] As described above, the smartphone 100 of this embodiment has basically the same configuration as the mobile terminal 100a of the first embodiment, and therefore has the same effects as the first embodiment.

[0085] Furthermore, according to this embodiment, when the smartphone 100 is in a stationary state, each time the in-camera 211 captures a user image 310 at a predetermined time interval, the display size of the AR object 410 is changed in accordance with a change in the size of the face image in the user image 310. Therefore, according to this embodiment, the display of the AR object 410 can be changed in accordance with a natural human behavior of moving the face closer when an object being viewed is too small and moving the face farther away when an object being viewed is too large.

[0086] Furthermore, according to this embodiment, a low-load process is used in which the movement of the user 300 is detected by simple image processing and the display parameters of the AR object 410 are changed according to the ratio. Therefore, the display size of the AR object 410 can be changed almost in real time in response to the movement of the user 300. In other words, there is little time lag until the display size changes, as occurs in processing using voice commands.

[0087] Therefore, when changing the display mode of the AR object 410, an intuitive and responsive user interface can be provided in response to the actions of the user 300.

[0088] Therefore, according to this embodiment, the AR object 410 can be displayed in the desired display mode on the display 212 of the smartphone 100 through simple processing without adding any new hardware, thereby improving usability for the user.

[0089] Third Embodiment Next, a third embodiment will be described. In this embodiment, when there are a plurality of AR objects 410, a function for selecting the AR object 410 whose display mode is to be adjusted is provided.

[0090] The basic configuration of this embodiment is the same as that of the second embodiment, and therefore the following description will focus on the different configurations. Fig. 6 is a functional block diagram of the smartphone 100b of this embodiment and the functions realized by the AR app of this embodiment.

[0091] The smartphone 100b of this embodiment includes a selection receiving unit 170 in addition to the functions of the second embodiment.

[0092] The selection receiving unit 170 receives a selection of an AR object 410 for which the display mode (display size) is to be changed from among the plurality of AR objects 410. The selection is received, for example, by the user's line of sight.

[0093] Specifically, the selection receiving unit 170 analyzes the user image 310 acquired by the user image acquiring unit 150 and determines the line of sight of the user 300. Then, the selection receiving unit 170 determines the AR object 410 displayed in the line of sight to be the selected AR object 410.

[0094] The gaze direction is estimated from the positions of, for example, detected feature points around the eyes, such as the inner corners, outer corners, and pupils, which are necessary for gaze detection, in the user image 310. These feature points are detected using known techniques.

[0095] The selection receiving unit 170 determines the AR object 410 that includes an intersection between a plane defined on the display 212 and an extension of the identified line of sight direction as the selected AR object 410. Note that, to identify the AR object 410, for example, display parameters (display position, display size) of each AR object 410 on the latest display screen generated by the display processing unit 143 are used. Note that the display parameters (display position, display size) of the AR object 410 stored in the AR object storage unit 180 may also be used.

[0096] The selection receiving unit 170 notifies the display control unit 140 of information specifying the selected AR object 410. For example, if each AR object 410 is assigned an identification number that uniquely identifies it, the selection receiving unit 170 notifies the display control unit 140 of the identification number.

[0097] The display processing unit 143 identifies the AR object 410 from the information transmitted from the selection receiving unit 170, and applies the display change rate Rd determined by the display mode determination unit 142 to the identified AR object 410. The sizes of the other AR objects 410 are not changed.

[0098] The operations of the other components are basically the same as those in the second embodiment.

[0099] The AR object selection process by the selection receiving unit 170 is performed independently of the AR display control process. For example, the selection receiving unit 170 executes the process each time the user image acquisition unit 150 acquires a user image 310, and notifies the display control unit 140 of the result of the process each time.

[0100] In the display control unit 140, the display processing unit 143 changes the display parameters of the AR object 410 notified at the timing of processing at the display change rate Rd specified by the display mode determination unit 142 and displays it in a superimposed manner.

[0101] [AR Display Mode Control Processing] The flow of the AR display mode control processing of this embodiment is shown in Figure 7. As described above, the flow of the AR display mode control processing of this embodiment is basically the same as that of the second embodiment. Therefore, the following description will focus on differences from the processing of the second embodiment. That is, when generating a display screen, the display processing unit 143 identifies the AR object 410 whose display mode is to be changed.

[0102] Therefore, as shown in the figure, when the display mode determination unit 142 calculates the display change rate Rd, the display processing unit 143 identifies the AR object 410 whose display mode is to be changed (step S3111). Here, as described above, the identification is performed based on the information identifying the selected AR object 410 transmitted from the selection receiving unit 170.

[0103] Then, the display parameters of the AR object 410 are changed in accordance with the display change rate Rd, and a display screen is generated in which the AR object 410 is displayed superimposed on the captured image (step S2114).

[0104] As described above, the smartphone 100b of this embodiment has basically the same configuration as the mobile terminal 100a of each of the above embodiments, and therefore has the same effects as the above embodiments.

[0105] Furthermore, according to this embodiment, when a plurality of AR objects 410 are displayed, it is possible to change the display mode of only a desired AR object 410 without using a hand.

[0106] Therefore, according to this embodiment, the AR object 410 can be displayed in the desired display mode on the display 212 of the smartphone 100b through simple processing without adding any new hardware, thereby improving usability for the user.

[0107] In the present embodiment, the user 300 may be able to set whether or not to activate the selection receiving unit 170. When the selection receiving unit 170 is not activated, the display control unit 140 changes the display modes of all of the AR objects 410 at the calculated display change rate Rd.

[0108] <Specific Example> A specific example of how to use the mobile terminal 100a and the smartphones 100 and 100b according to the above-described embodiments will now be described. The following description will be given taking the smartphone 100 according to the second embodiment as an example.

[0109] 8A, in a store such as a supermarket, product information that cannot be displayed on a product display shelf or the like may be displayed in AR on the display 212 of the smartphone 100 carried by the user 300. In this case, an AR object 410 of the product information is displayed superimposed on a captured image 430 including an image of the product acquired by the outer camera 221.

[0110] In such a case, if the user 300 holds the smartphone 100 in one hand and the shopping cart in the other, the user 300 cannot perform a normal pinch operation, etc. In other words, the user 300 cannot change the size of the AR object 410 on the display 212 of the smartphone 100.

[0111] However, according to the smartphone 100 of each of the above embodiments, as shown in Fig. 8(b) , when the user 300 moves their face closer without moving the smartphone 100, the size of the face image in the user image 310 captured by the in-camera 211 increases. As a result, the AR object 410 of the product information is enlarged accordingly. Conversely, when the user 300 moves their face away and the size of the face image decreases, the AR object 410 is displayed in a reduced size accordingly.

[0112] Therefore, according to the smartphone 100 of each of the above embodiments, when viewing product information displayed in AR while both hands are occupied, for example, in a supermarket, the user can enlarge or reduce the AR display to the desired size simply by moving their face.

[0113] That is, even if the user 300 has both hands full, the size of the background product image remains the same, and only the AR-displayed product information can be enlarged or reduced. In this case, the product information is enlarged when the user brings their face closer and reduced when the user moves their face away, which is close to the gestures that the user 300 naturally makes and the resulting changes that the user expects. Therefore, the user 300 can accept this without feeling uncomfortable.

[0114] [Selfie] For example, a user may take a selfie with the AR object 410 using the smartphone 100. When taking a selfie, the user superimposes an AR character, which is the AR object 410, on a captured image 430 of the user himself / herself. Therefore, since the user 300 holds the smartphone 100 in one hand and strikes a pose with the other hand, the user 300 cannot change the AR object 410 by a normal pinch operation or the like.

[0115] However, according to the smartphone 100 of each of the above embodiments, the user 300 first activates an AR app set for self-portraits from among the AR apps of each of the above embodiments. Then, as shown in Fig. 9A, the user 300 fixes the smartphone 100 and moves his / her face closer to or farther away from the smartphone 100. This allows the size of the AR object 410 to be adjusted, as shown in Fig. 9B.

[0116] Note that an actual selfie (photograph) is taken by positioning the smartphone 100 at a desired distance after adjusting the size of the AR object 410. Therefore, even if the user moves their face forward or backward to adjust the size of the AR object 410, this does not affect the finished photo.

[0117] In the AR application for taking selfies, the captured image acquisition unit 160 activates the inner camera 211 .

[0118] [Waterside / Cold Climate] For example, there is an event that utilizes the AR object 410 at a waterside, such as observing marine life by the pool. When operating the smartphone 100 at a waterside, the user's hands are wet and the user cannot touch the display 212. In other words, the user 300 cannot change the display mode of the AR object 410 by pinching or the like.

[0119] For example, the AR object 410 may be used to provide route guidance in cold regions such as ski resorts. In cold regions, the user 300 is wearing gloves and is therefore unable to touch the display 212. In other words, it is difficult for the user 300 to change the display mode of the AR object 410 by performing a pinch operation or the like.

[0120] In such a case, according to the smartphone 100 of each of the above embodiments, for example, first, the AR app of each of the above embodiments is activated by a voice command or the like. Then, with the smartphone 100 fixed, the distance between the smartphone 100 and a predetermined part is changed to change the image size of the predetermined part in the user image 310. This allows the size of the AR object 410 to be adjusted, for example, as shown in FIG. 9C .

[0121] [Barrier-Free Guidance] For example, various kinds of guidance may be provided using the AR object 410 in airports, large shopping malls, etc. For example, people who cannot operate the smartphone 100 using their fingers, such as people with injuries to their arms or hands or people with physical disabilities, have difficulty adjusting the display mode of the AR object 410.

[0122] However, according to the smartphone 100 of each of the above embodiments, as described above, the AR object 410 can be adjusted to the desired display mode without any finger operation, so that services utilizing AR objects can be provided comfortably to more people.

[0123] <Modification 1> In the above embodiments, the face is used as the part of the user 300 whose size is to be specified, but this is not limiting. For example, the pupils of the eyes may be used. Also, various facial features such as the eyes, nose, mouth, and their arrangement may be used.

[0124] In particular, when using the pupils of the eyes, the method of changing the size thereof is not limited to moving the position of the face back and forth, but may also be, for example, closing the eyes, etc. In particular, when the selection of the AR object 410 is accepted by gaze as in the third embodiment, it is advantageous that the selection and the part size specification can be performed on the same part.

[0125] Furthermore, in the above examples, for example, when near water or in cold regions, the screen cannot be operated with bare hands because the hands are wet or wearing gloves. In other words, the hands are not necessarily occupied. Therefore, in these cases, the predetermined part may be, for example, a wet hand or a gloved hand.

[0126] <Modification 2> In the above embodiments, the display size has been described as an example of a display aspect (display parameter) to be changed, but this is not limiting. For example, contrast, brightness, etc. may also be changed. The display processing unit 143 multiplies the default values ​​of these display parameters by the calculated display change rate Rd to determine the contrast and brightness to be displayed.

[0127] The parameter to be changed is not limited to one of the display size, contrast, and brightness of the AR object 410. For example, two or more parameters may be changed.

[0128] <Modification 3> In addition, in each of the above embodiments, the part size change rate calculation unit 141 calculates the part size change rate Rb, the display mode determination unit 142 calculates the display change rate Rd proportional to it, and the display processing unit 143 changes the display parameters using this display change rate Rd, but this is not limited to this. For example, the display change rate Rd may be set as a discrete value for each predetermined range of the part size change rate Rb.

[0129] For example, as shown in FIG. 10, the display change rate Rd may be determined in advance for each range of the region size change rate Rb and stored as a table.

[0130] Furthermore, two types of changes, large and small, may be used. That is, when the part size becomes larger than the initial size, the display processing unit 143 enlarges the display parameters at a predetermined specific magnification. On the other hand, when the part size becomes smaller, the display processing unit 143 reduces the display parameters at a predetermined specific reduction rate.

[0131] Furthermore, the display parameters to be changed may be color or display position. In this case, for example, the color to be used, the amount of displacement of the display position, the direction of displacement, etc. are determined in advance for each part size change rate Rb and stored as a table. For example, the closer the face is brought, the closer the face is displayed to the center of the display 212.

[0132] Note that a predetermined tolerance may be set for determining the change in region size. For example, if the rate of change from the initial size is within 10%, it may be determined that there is no change (region size change rate 0%). In other words, region size change rate calculation unit 141 may output the determined region size change rate Rb as 0% if it is within a predetermined tolerance range.

[0133] <Modification 4> In the above embodiments, the captured image 430 of real space displayed on the display 212 is acquired by the in-camera 211 or the out-camera 221 provided in the mobile terminal (such as the smartphone 100), but is not limited to this. For example, the image may be acquired by an external camera capable of transmitting and receiving data to and from the mobile terminal (such as the smartphone 100), such as a wearable camera or a camera mounted on a drone.

[0134] <Modification 5> In each of the above embodiments, the display processing unit 143 changes the values ​​of the display parameters of the AR object 410 in accordance with a change in the size of a predetermined part of the user 300 in the user image. However, this is not limited to this.

[0135] For example, a known facial image processing program may be used to determine the facial orientation of the user 300, and the display processing unit 143 may accordingly change the display parameter values ​​of the AR object 410. Furthermore, if the AR object 410 is a three-dimensional object, the display orientation may be configured to change depending on the orientation of the facial image.

[0136] Furthermore, the display processing unit 143 may move the display position of the AR object 410 according to the direction of the face of the user 300. Specifically, when the user 300 turns his / her face to the right, the display position of the AR object moves to the right, for example, to follow the movement of the user 300.

[0137] <Variation 6> The timing of registering the initial size is not limited to the methods described in the above embodiments. For example, the initial size registration unit 110 may register the part size that is first output after receiving a still signal as the initial size. The initial size may also be registered in accordance with an instruction from the user 300. The instruction from the user 300 may be in any of various ways that can be detected by the smartphone 100. For example, the instruction may be a voice command, a predetermined action (such as a gesture) by the user 300, or a combination thereof.

[0138] For example, in the case of a predetermined action (gesture, etc.) by user 300, initial size registration unit 110 analyzes, at predetermined time intervals, an image captured by outer camera 221 or inner camera 211. Then, the part size identified by part size identification unit 120 at the timing when the predetermined action is extracted is set as the initial size.

[0139] In the case of a voice command or the like, the voice acquired by a microphone 997 (described later) is analyzed by the initial size registration unit 110. Then, at the timing when predetermined voice data is detected, the body part size specified by the body part size specification unit 120 is set as the initial size.

[0140] <Variation 7> Furthermore, the selection receiving unit 170 of the third embodiment analyzes the gaze direction of the user 300 and receives the selection of the AR object 410 for which the display mode is adjusted, but the selection receiving method is not limited to this. Any method for receiving various instructions from the user 300 by the smartphone 100 can be used. For example, a voice command or a gesture by the user may be used. Furthermore, the gaze direction, the voice command, the gesture, etc. may be combined.

[0141] For example, when a selection is accepted by a voice command or the like, the selection accepting unit 170 analyzes the voice acquired by a microphone 997 (described later) to identify the AR object 410 selected by the user 300. When a selection is accepted by a gesture of a hand or the like, the selection accepting unit 170 analyzes an image captured by the outer camera 221 or the inner camera 211 to identify the AR object 410 selected by the user 300.

[0142] <Modification 8> In the above embodiment, the mobile terminal is described as being the smartphone 100 or 100b, but the mobile terminal is not limited to being the smartphone 100 or 100b. For example, the mobile terminal may be any portable information processing device such as a notebook PC (Personal Computer) or a tablet terminal.

[0143] [Hardware Configuration] An example of the hardware configuration of the mobile terminal 100a and the smartphones 100 and 100b (hereinafter, represented by the smartphone 100) will be described below. FIG.

[0144] As shown in this figure, in addition to the display 212, in-camera 211, and out-camera 221 described above, the smartphone 100 also includes a CPU (Central Processing Unit) 991, a main storage device (memory) 992, an auxiliary storage device 993, a communication device 994, a sensor 995, an operation device 996, a microphone 997, and a speaker 998, which are interconnected by an internal bus.

[0145] The CPU 991, for example, loads a program stored in the auxiliary storage device 993 into the main storage device 992 and executes it to realize the above-described functions and to comprehensively control the entire smartphone 100. Note that one or more processors such as an MPU (Micro Processing Unit) may be used instead of the CPU 991.

[0146] The main storage device 992 is a memory such as a RAM (Random Access Memory), and is a work area used by the CPU 991 when processing, for example, an AR application program.

[0147] The auxiliary storage device 993 is, for example, a read-only memory (ROM), a hard disk drive (HDD), or a solid state drive (SSD). The auxiliary storage device 993 stores various programs executed by the smartphone 100, including an AR app. In this embodiment, the initial size storage unit 190 and the AR object storage unit 180 are configured in the auxiliary storage device 993. Various data, tables, and the like used during processing are also stored in the auxiliary storage device 993, for example.

[0148] The programs stored in the auxiliary storage device 993 can be provided as program products recorded on a non-transitory computer-readable storage medium. The auxiliary storage device 993 can be used to store various programs recorded on the non-transitory computer-readable storage medium for the medium to long term.

[0149] The communication device 994 communicates with external devices. For example, it is configured with an IC chip or the like and includes a communication circuit, an antenna, etc. For example, it includes a LAN communication unit that performs data communication with a LAN, a telephone line communication unit that performs wireless communication with a base station of a mobile telephone communication network, and a short-range wireless communication unit that performs short-range wireless communication using a method such as Bluetooth (registered trademark) or infrared.

[0150] The sensor 995 is a group of various sensors included in the smartphone 100. For example, the sensor 995 includes an acceleration sensor, a gyro sensor, a geomagnetic sensor, a GPS receiver, and the like.

[0151] The operation device 996 receives operation inputs to the smartphone, such as turning the power on / off, etc. The operation device 996 includes hardware buttons, touch sensors, etc.

[0152] The microphone 997 collects sounds around the smartphone 100, converts them into data, and passes them to other functional units. The speaker 998 outputs processed audio signals to the outside.

[0153] The above-described functions of the smartphone 100 are realized by the CPU 991 loading a program (AR application) stored in the auxiliary storage device 993 into the main storage device 992 and executing the program.

[0154] The hardware configuration of the smartphone 100 is not limited to this. The smartphone 100 may include hardware not shown. For example, the smartphone 100 may include an expansion interface to which an external display device, input device, storage device, etc. can be connected.

[0155] The AR application may be recorded on a computer-readable storage medium, which may be a non-transitory storage medium such as a semiconductor memory, a hard disk, a magnetic recording medium, or an optical recording medium. The present invention may also be embodied as a computer program product.

[0156] Furthermore, each function of the smartphone 100 may be implemented, for example, by an integrated circuit (IC) dedicated to each process, an application specific integrated circuit (ASIC), a system on chip (SOC), a field programmable gate array (FPGA), or the like.

[0157] In the process flow used in the above explanation, multiple steps (processes) are described in order, but the order in which each step is performed is not limited to the order described. For example, the order of the steps shown in the figure can be changed to the extent that the content is not affected, such as by performing each process in parallel.

[0158] Although the embodiments and modifications of the present invention have been described above, the present invention is not limited to the above-described embodiments and can be modified in various ways that would be understandable to those skilled in the art. Each embodiment and modification can be combined with other embodiments as appropriate. Furthermore, for example, the network configurations and element configurations shown in the drawings are examples intended to aid in understanding the present invention, and the present invention is not limited to the configurations shown in these drawings.

[0159] Finally, preferred embodiments of the present invention will be summarized. Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A mobile terminal includes: a body part size determination unit configured to extract body part images of predetermined body parts of a user from a user image, which is an image of the user, and determine the size of the extracted body part image as a body part size; a motion detection unit configured to detect that the device is in a stationary state; and a display control unit configured to control a display mode of an augmented reality object on a display, wherein the display control unit is configured to, during the stationary state, change the display mode of the augmented reality object to a predetermined mode in response to a change in the body part size from an initial size, which is a pre-registered initial value of the body part size. (Supplementary Note 2) In the mobile terminal described in Supplementary Note 1, it is preferable that the display control unit includes: a part size change rate calculation unit configured to calculate a part size change rate, which is the rate of change of the part size from the initial size; a display mode determination unit configured to calculate a display change rate, which is the rate of change of a display parameter of the augmented reality object from a default value, using the part size change rate; and a display processing unit configured to change the display parameter of the augmented reality object using the display change rate. (Supplementary Note 3) In the mobile terminal described in Supplementary Note 2, it is preferable that the display parameter is at least one of display size, contrast, brightness, color, and display position. (Supplementary Note 4) In the mobile terminal described in any of Supplements 1 to 3, it is preferable that the part is at least one of face, eyes, nose, mouth, and hands. (Supplementary Note 5) In the mobile terminal described in any of Supplements 1 to 4, it is preferable that the mobile terminal further includes a selection receiving unit configured to receive selection of the augmented reality object whose display mode is to be changed, and it is preferable that the display control unit is configured to change the display mode of the selected augmented reality object.(Supplementary Note 6) In the mobile terminal described in Supplementary Note 5, it is desirable that the selection receiving unit is configured to receive selection of the augmented reality object that changes the display mode in accordance with at least one of the user's line of sight, voice, and action instruction. (Supplementary Note 7) In the mobile terminal described in any of Supplements 1 to 6, it is desirable that the mobile terminal further comprises an initial size registration unit configured to register the initial size, and the part size identification unit is configured to identify the part size each time the user image is acquired at a predetermined time interval, and the initial size registration unit is configured to register the part size identified using the user image acquired first as the initial size. (Supplementary Note 8) In the mobile terminal described in any of Supplements 1 to 7, it is desirable that the mobile terminal further comprises an in-camera, and the user image is configured to be acquired by the in-camera. (Supplementary Note 9) An augmented reality object display control method on a mobile terminal includes: extracting part images of predetermined parts of a user from an image captured of the user; specifying the size of the extracted part image as a part size each time the part is extracted; and, while the mobile terminal is stationary, changing a display mode of the augmented reality object on a display of the mobile terminal in accordance with a change in the specified part size from an initial size that is a pre-registered initial value for the part size. (Supplementary Note 10) A program causes a computer mounted on the mobile terminal to function as: a part size specification unit that extracts part images of predetermined parts of the user from an image captured of the user and specifies the size of the extracted part image as the part size; a motion detection unit that detects that the device is in a stationary state; and an augmented reality display control unit that controls, while the mobile terminal is stationary, to change the display mode of the augmented reality object on the display to a predetermined mode in accordance with a change in the part size from an initial size that is a pre-registered initial value for the part size.(Supplementary Note 11) In the mobile terminal described in any of Supplements 1 to 8, it is desirable that the display control unit is configured to calculate a part size change rate, which is a rate of change of the part size from the initial size, and change and display the display parameters of the augmented reality object according to a display change rate predetermined for each part size change rate. (Supplementary Note 12) In the mobile terminal described in Supplementary Note 5, it is desirable that the selection receiving unit is configured to detect the user's line of sight by analyzing the user image, and select the augmented reality object displayed in an area on the display that intersects with the line of sight as the selected augmented reality object. Note that Supplements 9 and 10 can be expanded into Supplements 2-8, 11, and 12, as in Supplementary Note 1.

[0160] The disclosures of the above-mentioned patent documents and other documents are incorporated herein by reference. Modifications and adjustments of the embodiments and variations are possible within the scope of the entire disclosure of the present invention (including the claims), and further based on the basic technical concepts thereof. Furthermore, various combinations and selections of the various disclosed elements (including each element of each claim, each element of each embodiment or variation, each element of each drawing, etc.) are possible within the scope of the disclosure of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible by a person skilled in the art in accordance with the entire disclosure and technical concepts, including the claims. In particular, with regard to the numerical ranges described herein, any numerical value or subrange included within the range should be construed as being specifically described, even if not otherwise specified.

[0161] 100: Smartphone, 100a: Portable terminal, 100b: Smartphone, 110: Initial size registration unit, 120: Part size identification unit, 120a: Part size identification unit, 130: Action detection unit, 130a: Action detection unit, 140: Display control unit, 140a: Display control unit, 141: Part size change rate calculation unit, 142: Display mode determination unit, 143: Display processing unit, 150: User image acquisition unit, 160: Captured image acquisition unit, 170: Selection reception unit, 180: AR object storage unit, 190: Initial size storage unit, 190a: Initial size storage unit, 210: First surface, 211: In-camera, 212: Display, 220: Second surface, 221: Out-camera, 300: User, 310: User image, 311: Face image, 312: Face image, 410: AR object, 411: default size, 412: size, 430: captured image, 991: CPU, 992: main storage device, 993: auxiliary storage device, 994: communication device, 995: sensor, 996: operation device, 997: microphone, 998: speaker, Rb: part size change rate, Rd: display change rate

Claims

1. A mobile terminal comprising: a body part size determination unit configured to extract body part images of predetermined body parts of a user from a user image, which is an image of the user, and determine the size of the extracted body part image as the body part size; a motion detection unit configured to detect that the device is in a stationary state; and a display control unit configured to control the display manner of an augmented reality object on a display, wherein the display control unit is configured to change the display manner of the augmented reality object to a predetermined manner in accordance with a change in the body part size from an initial size, which is a pre-registered initial value of the body part size, during the stationary state.

2. A mobile terminal as described in claim 1, wherein the display control unit comprises: a part size change rate calculation unit configured to calculate a part size change rate, which is the rate of change of the part size from the initial size; a display mode determination unit configured to calculate a display change rate, which is the rate of change of the display parameters of the augmented reality object from default values, using the part size change rate; and a display processing unit configured to change and display the display parameters of the augmented reality object using the display change rate.

3. A mobile terminal according to claim 2, wherein the display parameter is at least one of display size, contrast, and brightness.

4. A mobile terminal according to any one of claims 1 to 3, wherein the part is a face.

5. A mobile terminal according to any one of claims 1 to 4, further comprising a selection receiving unit configured to receive a selection of the augmented reality object whose display mode is to be changed, and the display control unit is configured to change the display mode of the selected augmented reality object.

6. A mobile terminal according to claim 5, wherein the selection receiving unit is configured to receive a selection of the augmented reality object that changes the display mode based on at least one of the user's line of sight, voice, and user's action instruction.

7. A mobile terminal as claimed in any one of claims 1 to 6, further comprising an initial size registration unit configured to register the initial size, wherein the part size determination unit is configured to determine the part size each time the user image is acquired at a predetermined time interval, and the initial size registration unit is configured to register the part size determined using the user image acquired first as the initial size.

8. A mobile terminal according to any one of claims 1 to 7, comprising an in-camera, and configured so that the user image is acquired by the in-camera.

9. An augmented reality object display control method in a mobile device, which extracts a part image of a predetermined part of a user from an image taken of the user, and each time an image is extracted, identifies the size of the extracted part image as the part size, and while the mobile device is stationary, changes the display manner of the augmented reality object on the display of the mobile device in accordance with the change in the identified part size from an initial size, which is the initial value of the part size that is registered in advance.

10. A program for causing a computer installed in a mobile terminal to function as: a body part size determination unit that extracts body part images of predetermined body parts of a user from an image of the user and determines the size of the extracted body part image as the body part size; a motion detection unit that detects whether the device is in a stationary state; and an augmented reality display control unit that controls the display manner of an augmented reality object on the display to change to a predetermined manner in accordance with changes in the body part size from an initial size, which is the initial value of the body part size that has been registered in advance, while the device is in the stationary state.

Images