Information processor and program

Abstract

To make it possible to respond to an inquiry about a character placed in a virtual space based on the movement of a terminal.SOLUTION: A program causes a computer to display, on a display unit included in a terminal, at least a part of a virtual space in which an object is placed, output an inquiry about predetermined processing, accept an input for the inquiry based on the movement of the terminal when at least a part of the object is displayed on the display unit, and perform the predetermined processing based on the input for the inquiry.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing device and a program. [Background technology]

[0002] Conventionally, a virtual reality (VR) technology has been put to practical use, which displays a virtual space on a mobile information terminal such as a smartphone and presents it to a user. In recent years, an augmented reality (AR) technology has been attracting attention, which displays artificial characters or structures, or animals, plants, or structures captured by imaging or the like on a mobile information terminal and presents them to a user when the mobile information terminal is pointed at a specific location. Hereinafter, the artificial characters or structures, or the captured images will be collectively referred to as objects.

[0003] For example, Patent Document 1 discloses an information processing device and a program relating to a technique for manipulating an object arranged in a virtual space by a user's input operation through a touch operation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-176764 A Summary of the Invention [Problem to be solved by the invention]

[0005] One of the objectives of the present invention is to enable a response to an inquiry about an object placed in a virtual space based on the movement of a terminal. [Means for solving the problem]

[0006] The program causes a computer to perform the following operations: displaying at least a portion of a virtual space in which an object is placed on a display unit included in a terminal; outputting a query regarding a specified process; accepting input in response to the query based on the movement of the terminal when at least a portion of the object is displayed on the display unit; and executing the specified process based on the input in response to the query.

[0007] The computer may be caused to set one of a plurality of operating modes to the object, and when a predetermined mode is set to the object and a query is output from the object, a predetermined process is executed.

[0008] The computer may be caused to place a plurality of objects in a virtual space, set one of a plurality of operation modes corresponding to each of the plurality of objects, accept input in response to a query based on the movement of the terminal, and execute a predetermined process based on the input in response to the query.

[0009] The movement of the terminal may be determined from an image obtained by combining at least a portion of the virtual space in which the object is placed with an image captured by the imaging unit.

[0010] When the computer is caused to execute a predetermined process based on the movement of the terminal, the image captured by the imaging unit may be fixed.

[0011] The predetermined process may be executed by the computer in such a manner that the process is executed based on the movement of the terminal when an amount of change in a parameter indicating the movement of the terminal is within a predetermined range.

[0012] The information processing device includes a display control unit that causes a terminal to display at least a portion of a virtual space in which objects are arranged, a query output unit that outputs a query regarding a specified process, and an execution unit that receives input in response to the query based on movement of the terminal when at least a portion of the objects displayed on the terminal are displayed, and executes the specified process based on the input in response to the query. Effect of the Invention

[0013] According to one embodiment of the present invention, a response to a query about an object placed in a virtual space can be made based on the movement of the terminal. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an information processing system according to an embodiment of the present invention; [Diagram 2] 1A is a block diagram showing a hardware configuration of an information processing device according to an embodiment of the present invention, and FIG. 1B is a block diagram showing a functional configuration of the information processing device according to an embodiment of the present invention. [Diagram 3] FIG. 1 is a diagram showing a virtual space in which objects are arranged according to an embodiment of the present invention. [Figure 4] 1A, 1B, 1C, and 1D are plan views showing a configuration of an information processing device according to an embodiment of the present invention. [Diagram 5] 1A, 1B, 1C, and 1D are perspective views showing the movement of an information processing device according to one embodiment of the present invention. [Figure 6] 4 is a flow chart illustrating a processing method according to an embodiment of the present invention. [Figure 7] 4 is a flow chart illustrating a processing method according to an embodiment of the present invention. [Figure 8] FIG. 1 is a diagram showing a virtual space in which a plurality of objects are arranged according to an embodiment of the present invention. [Figure 9] 4 is a flow chart illustrating a processing method according to an embodiment of the present invention. [Figure 10] 1A is a plan view showing a virtual space in which an object is placed according to an embodiment of the present invention, FIG. 1B is a plan view showing a captured image according to an embodiment of the present invention, and FIG. 1C is a plan view showing a virtual space and a captured image superimposed according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention can be implemented in many different aspects, and should not be construed as being limited to the description of the embodiment exemplified below. Furthermore, in this specification and each figure, elements similar to those described above with respect to the previous figures may be given the same reference numerals (or reference numerals with A, B, -1, -2, etc. suffixed thereto) and detailed description may be omitted as appropriate. Note that the letters "first" and "second" suffixed to each element are convenient labels used to distinguish each element, and have no further meaning unless otherwise specified. 1. First embodiment 1-1. Information processing system

[0016] 1 is a block diagram showing an information processing system 1 according to an embodiment of the present invention. The information processing system 1 includes a server 20, a plurality of information processing devices 10, and a network 40 such as a communication line.

[0017] As an example, when a user enters a room, the information processing system 1 according to an embodiment of the present invention can control whether or not to turn on the lights in the room, based on an inquiry from an object placed in a virtual space. An overview of the above example will be described. An object placed in the virtual space is set to a standby mode and an inquiry mode. The object is usually set to the standby mode. When the user moves the information processing device 10 in response to the inquiry from the object, the information processing device 10 or the object executes whether or not to turn on the lights.

[0018] In one embodiment of the present invention, the operation mode indicates a state of the information processing device 10 or an object, such as a standby mode, a query mode, etc. Here, the standby mode is a mode in which the information processing device 10 or the object waits for an instruction (input) from a user or the server 20, and is also a mode in which the user normally uses the information processing device 10, such as for communication such as a call or a network connection, or for imaging. Also, the query mode is, for example, a mode in which the object queries, asks questions, or engages in a dialogue with the user.

[0019] In one embodiment of the present invention, the predetermined process is, for example, a series of processes for turning on the lights. Hereinafter, the predetermined process will be described taking as an example control of whether or not to turn on the lights in a room.

[0020] In FIG. 1, the server 20 is a computer that provides a program to the information processing device 10 via the network 40. The server 20 is also a computer that provides various services to the information processing device 10 via the network 40. The various services may include, for example, a social networking service (SNS) and the provision of installation files for various applications. For example, when the information processing device 10 inquires about the provision of an application or program to the server 20, the server 20 may provide the installation file or program of the application to the information processing device 10. A user can connect to the server 20 via the network 40 by installing an application in the information processing device 10 and using the application installed in the information processing device 10. The information processing device 10 executes a predetermined process based on the provided program. Note that, in FIG. 1, the server 20 is illustrated as one device, but the server 20 may be composed of multiple devices. The server 20 may also provide objects, virtual spaces, imaging, inquiry output, and other functions. It may also have a storage device for storing information such as image or video information (Is it OK?, Yes or No), programs, and parameters required for the operation of the programs.

[0021] The information processing device 10 is, for example, a mobile information terminal such as a smartphone or a tablet, a portable game machine, or a personal computer. Note that the multiple information processing devices 10 can communicate with each other. 1-2. Hardware Configuration 100 of the Information Processing Device 10

[0022] 2A is a diagram showing a hardware configuration 100 of an information processing device 10 according to an embodiment of the present invention. The hardware configuration 100 is included in a mobile information terminal. The hardware configuration 100 includes a communication unit 101, an operation unit 103, a display unit 105, an imaging unit 107, a control unit 109, a storage unit 111, and a detection unit 113.

[0023] The communication unit 101 is a communication interface included in the information processing device 10. In one embodiment of the present invention, the information processing device 10 is connected to the server 20 via the network 40 by way of the communication unit 101. The information processing device 10 transmits and receives information to and from the server 20. The information processing device 10 may also be connected to another information processing device 10 via the network 40. In one embodiment of the present invention, information may be replaced with data.

[0024] The operation unit 103 is a user interface such as, for example, a button or a touch sensor. When the operation unit 103 detects that a user operation has been input to the interface, it outputs various control signals corresponding to the input of the user operation to the control unit 109. In one embodiment of the present invention, the operation unit 103 is, for example, a touch sensor.

[0025] The display unit 105 is provided integrally within a housing constituting the information processing device 10. The display unit 105 is, for example, a liquid crystal display panel or an EL (Electroluminescence) display panel. The display unit 105 displays a partial area of ​​the virtual space in which an object is arranged. Although details will be described later, one object may be arranged in the virtual space, or multiple objects may be arranged. For example, as shown in FIG. 3, an object 300 is arranged in a partial area 310A of the virtual space 305. Moreover, an object 300 is not arranged in a partial area 310B of the virtual space 305.

[0026] The imaging unit 107 has an imaging element that converts light into an electric signal, and includes, for example, a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor. The imaging unit 107 is, for example, a camera. The imaging unit 107 is driven based on an imaging control signal output from the control unit 109. The imaging unit 107 outputs captured image information or video information to the control unit 109, the storage unit 111, or the server 20.

[0027] The control unit 109 has an arithmetic processing circuit and executes a program using the arithmetic processing circuit. In this specification and the like, the arithmetic processing circuit is a circuit capable of arithmetic processing, such as a processor, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The control unit 109 may receive a program stored in the server 20 via the communication unit 101.

[0028] The storage unit 111 is a storage device capable of permanently storing information and rewriting information. The storage unit 111 is, for example, a non-volatile memory. The storage unit 111 stores information such as objects, virtual spaces, imaging, image information or video information such as inquiry outputs (Are you okay?, Yes or No), information regarding the operation mode set for the object, programs, and parameters required in the operation of the programs. The storage unit 111 may have a rewritable storage device used for temporarily storing information. The rewritable storage device is, for example, a volatile memory, SRAM (Static Random Access Memory), or DRAM (Dynamic Random Access Memory). The storage unit 111 may be used as a storage area for storing the expansion area of the program stored in the storage unit 111 and intermediate data such as information output in the operation of the program.

[0029] The detection unit 113 detects the movement of the information processing device 10 and receives an input for an inquiry. Specifically, when an inquiry output signal is output from the inquiry output unit 205, the detection unit 113 detects an instruction input by the user to the information processing device 10. Detecting an instruction input by the user to the information processing device 10 by the detection unit 113 may also be described as detecting the user's response or detecting the movement given by the user to the information processing device 10. The detection unit 113 has a sensor capable of detecting the movement given by the user to the information processing device 10, such as an angular velocity sensor (gyro sensor), a geomagnetic sensor, or an acceleration sensor. For example, when the detection unit 113 has an angular velocity sensor, the detection unit 113 detects, with the angular velocity sensor, the movement of the user moving the information processing device 10 up and down when an inquiry output signal is output from the inquiry output unit 205. 1-3. Functional configuration 200 of the information processing device 10

[0030] 2B is a diagram showing a functional configuration 200 of the information processing device 10 that executes a program according to an embodiment of the present invention. The functional configuration 200 is included in the control unit 109. The functional configuration 200 includes a display control unit 201, a setting unit 203, a query output unit 205, and an execution unit 209.

[0031] The setting unit 203 has a function of setting one of a plurality of operation modes for an object. The setting unit 203 receives an instruction (a setting control signal) to set an operation mode from the operation unit 103 by a user's touch operation on the operation unit 103. When the setting unit 203 receives the setting control signal, it sets the standby mode and the query mode for the object. Furthermore, when the setting unit 203 receives a query output signal from the query output unit 205, it transitions the state of the object from the standby mode to the query mode. When the setting unit 203 transitions the object to the query mode, it outputs a transition signal to the query output unit 205.

[0032] The inquiry output unit 205 has a function of outputting an inquiry regarding a predetermined process to the display unit 105. The inquiry output unit 205 outputs an inquiry output signal to the display control unit 201, the setting unit 203, and the operation determination unit 207. For example, when a user enters a room, a human presence sensor provided in the room reacts, and the human presence sensor outputs a sensor signal to the server. When the server receives the sensor signal, it outputs a detection signal to the inquiry output unit 205. The inquiry output unit 205 receives the detection signal from the server. Furthermore, when the inquiry output unit 205 receives the detection signal from the server, it outputs an inquiry output signal (a signal as to whether or not to turn on the light) to the display control unit 201, the setting unit 203, and the operation determination unit 207.

[0033] The motion determination unit 207 has a function of determining an input in response to an inquiry from a detection result of the motion of the information processing device 10 by the detection unit 113. Specifically, the motion determination unit 207 determines whether or not an input in response to the inquiry has been made from the motion of the information processing device 10 by the detection unit 113, and outputs the determination result to the execution unit 209. For example, when the motion determination unit 207 determines that an input in response to the inquiry has been made from the motion of the information processing device 10 by the detection unit 113, it outputs an input detection signal to the execution unit 209. For example, when the motion determination unit 207 determines that an input in response to the inquiry has not been made from the motion of the information processing device 10 by the detection unit 113, it outputs an input non-detection signal to the execution unit 209. For example, an input detection signal or an input non-detection signal for determining an input in response to an inquiry from the up, down, left, and right motion of the information processing device 10 may be processed by the information processing device 10, and an input detection signal or an input non-detection signal for determining an input in response to an inquiry from the tilt of the information processing device 10 may be processed by the server 20.

[0034] The execution unit 209 has a function of receiving an input detection signal or an input non-detection signal output from the operation determination unit 207 and executing a predetermined process. When the execution unit 209 receives an input detection signal or an input non-detection signal from the operation determination unit 207, it outputs a signal (execution control signal) for executing a predetermined process. For example, when the execution unit 209 receives a signal for turning on a light from the operation determination unit 207, When the (input detection signal) is received, an execution control signal is output to a switch that turns on the light in the room. Then, the light in the room is turned on. The execution unit 209 may output the execution control signal to the server 20. Then, when the server 20 receives the execution control signal, it may output a signal that controls the light in the room to the switch that turns on the light in the room, and the light in the room may be turned on.

[0035] The display control unit 201 has a function for displaying image information or video information on the display unit 105. Specifically, the display control unit 201 controls the image information or video information to be displayed on the display unit 105. For example, when the display control unit 201 receives a display control signal from the operation unit 103, the display control unit 201 outputs image information related to a partial area of ​​the virtual space in which an object is placed, and image information or video information captured by the imaging unit 107, to the display unit 105. Also, for example, when a signal as to whether or not to turn on the light is output from the inquiry output unit 205, the display control unit 201 outputs an inquiry output (is it OK?, yes or no, etc.) to the display unit 105.

[0036] Here, the server 20 may include a part of the functional configuration 200 and execute a part of the functions of the functional configuration 200. 1-4. Output of a Query

[0037] An output of a query according to an embodiment of the present invention will be described with reference to Fig. 4. Fig. 4(A), Fig. 4(B), Fig. 4(C) and Fig. 4(D) are plan views showing a configuration of an information processing device 10 according to an embodiment of the present invention. Descriptions of configurations that are the same as or similar to those in Figs. 1 to 3 may be omitted.

[0038] As shown in Figures 4(A), 4(B), 4(C), and 4(D), the information processing device 10 has a display unit 105 and an imaging unit 107. In Figures 4(A), 4(B), 4(C), and 4(D), descriptions of hardware configurations other than the display unit 105 and the imaging unit 107 are omitted.

[0039] 4(A), the display unit 105 displays a partial area 310A of the virtual space in which the object 300 is placed. The display unit 105 may display a part of the object 300 in the partial area 310A of the virtual space. For example, the display unit 105 may display the part of the object 300 above the waist, or the face of the object 300.

[0040] Further, FIG. 4(A) is a diagram showing that when the display control unit 201 outputs a display control signal from the operation unit 103, the display unit 105 outputs image information related to a partial region of the virtual space 305 where the object 300 is arranged. Further, FIG. 4(A) is also a diagram showing a state in which the object 300 has transitioned to the inquiry mode but is not outputting an inquiry.

[0041] As shown in FIG. 4(B), the display unit 105 may also display a partial region 320B of the virtual space where the object 300 is not arranged.

[0042] As shown in FIG. 4(C), in addition to the image in FIG. 4(A), the display unit 105 displays an inquiry output 330. FIG. 4(C) is a diagram showing that when the display control unit 201 outputs an inquiry output signal from the inquiry output unit 205, the display unit 105 outputs an inquiry output (Okay?)

[0043] As shown in FIG. 4(D), the display unit 105 may also display the inquiry output 330-2 shifted and superimposed with respect to the inquiry output 330-1. FIG. 4(D) is a diagram showing that when the display control unit 201 outputs an inquiry output signal twice from the inquiry output unit 205, the display unit 105 outputs an inquiry output (Okay?). 1-5. Input for Inquiry

[0044] With reference to FIGS. 5(A), 5(B), 5(C), and 5(D), the input for the inquiry according to an embodiment of the present invention will be described. FIGS. 5(A), 5(B), 5(C), and 5(D) are perspective views showing the movement of an information processing apparatus that executes a program according to an embodiment of the present invention. The description of the same or similar configurations as those in FIGS. 1 to 4 may be omitted. In this specification and the like, the input for the inquiry may also be referred to as the movement of the terminal. In FIGS. 5(A), 5(B), 5(C), and 5(D), the display surface 105A of the display unit 105 of the information processing apparatus 10 is simply referred to as the display surface 105A.

[0045] FIG. 5(A) is a diagram showing a case where an input for an inquiry is made by moving the information processing device 10 in a direction (up-down direction) approximately parallel to a normal line E approximately perpendicular to the left-right direction (AB direction) of the display surface 105A. FIG. 5(B) is a diagram showing a case where an input for an inquiry is made by moving the information processing device 10 in a direction (left-right direction) approximately parallel to a normal line F approximately perpendicular to the up-down direction (CD direction) of the display surface 105A. FIG. 5(C) is a diagram showing a case where an input for an inquiry is made by moving the information processing device 10 so as to be inclined with respect to a normal line G approximately perpendicular to the AB direction of the display surface 105A. FIG. 5(D) is a diagram showing a case where an input for an inquiry is made by moving the information processing device 10 so as to be inclined with respect to a normal line H approximately perpendicular to the CD direction of the display surface 105A. In this specification and the like, the left-right direction is indicated as a direction along A and B, and the up-down direction is indicated as a direction along C and D. Descriptions of configurations that are the same as or similar to those in FIG. 1 to FIG. 4 may be omitted.

[0046] As shown in FIG. 5(A), when the user moves the information processing device 10 up and down by a movement amount d1 within a range that satisfies a predetermined first condition, the detection unit 113 (FIG. 2(A)) detects the user's movement of the information processing device 10 up and down. When the detection unit 113 detects the movement of the information processing device 10, the motion determination unit 207 (FIG. 2(B)) outputs an input detection signal to the execution unit 209. On the other hand, when the user moves the information processing device 10 up and down by a movement amount d1 within a range that does not satisfy the predetermined first condition, when the detection unit 113 detects the movement of the information processing device 10, the motion determination unit 207 outputs an input non-detection signal to the execution unit 209. The execution unit 209 prompts the user to input the inquiry again. As a result, for example, the display unit 105 displays an image similar to the inquiry output shown in FIG. 4(C). Also, as shown in FIG. 4(D), the display unit 105 may further shift and display an inquiry output 330-2 over the inquiry output 330-1.

[0047] 5(B), when the user moves the information processing device 10 left and right by a movement amount d2 within a range that satisfies the predetermined second condition, the detection unit 113 detects the movement of the information processing device 10 and the motion determination unit 207 outputs an input detection signal to the execution unit 209. On the other hand, when the user moves the information processing device 10 left and right by a movement amount d2 within a range that does not satisfy the predetermined second condition, the detection unit 113 detects the movement of the information processing device 10 and the motion determination unit 207 outputs an input non-detection signal to the execution unit 209. The execution unit 209 again prompts the user to input in response to the inquiry.

[0048] 5(C), when the user tilts the information processing device 10 within a range in which the angle θx satisfies the predetermined third condition, the detection unit 113 detects the movement of the information processing device 10 and the movement determination unit 207 outputs an input detection signal to the execution unit 209. On the other hand, when the user tilts the information processing device 10 within a range in which the angle θx does not satisfy the predetermined third condition, the detection unit 113 detects the movement of the information processing device 10 and the movement determination unit 207 outputs an input non-detection signal to the execution unit 209. The execution unit 209 prompts the user to enter an input in response to the inquiry again.

[0049] 5(D), when the user tilts the information processing device 10 within a range in which the angle θy satisfies the predetermined fourth condition, the detection unit 113 detects the movement of the information processing device 10 and the movement determination unit 207 outputs an input detection signal to the execution unit 209. On the other hand, when the user tilts the information processing device 10 within a range in which the angle θy does not satisfy the predetermined fourth condition, the detection unit 113 detects the movement of the information processing device 10 and the movement determination unit 207 outputs an input non-detection signal to the execution unit 209. The execution unit 209 again prompts for input in response to the inquiry.

[0050] For example, when an input is made in response to a query, the driving of the imaging unit 107 is stopped. Also, when an input is made in response to a query, the image captured by the imaging unit 107 and displayed on the display unit 105 may be fixed, or the image captured by the imaging unit 107 may no longer be displayed on the display unit 105. This makes it possible to distinguish between the user's normal use of the information processing device 10 and the user's input in response to a query about an object by moving the information processing device 10. Therefore, malfunctions when the user makes an input in response to a query about an object are suppressed. 1-6. Processing method for executing a program

[0051] A processing method according to an embodiment of the present invention will be described with reference to Figures 1, 2, 4, 5, 6 and 7. Figures 6 and 7 are flowcharts showing a processing method according to an embodiment of the present invention. Descriptions of configurations that are the same as or similar to those in Figures 1 to 5 may be omitted.

[0052] When the information processing device 10 executes a program, it starts a predetermined process. For example, the information processing device 10 starts an application in response to a user's instruction. First, the setting unit 203 (FIG. 2(B)) sets a standby mode and a query mode for an object (step 501 (S501)).

[0053] In step 503 (S503), for example, when the user touches the operation unit 103, the information processing device 10 reads out image data of the object and the virtual space, for which one of a plurality of operation modes has been set, from the server 20 to the display control unit 201 (FIG. 2(B)). The display control unit 201 places the object in the virtual space based on the read image data of the object and the virtual space. The object is in a state (standby mode) waiting for an instruction (input) from the user or the server 20. This is step 503 (S503).

[0054] In step 505 (S505), if there is an input from the user or server 20 (Yes), the setting unit 203 (FIG. 2(B)) transitions the object from standby mode to query mode. If there is no input from the user or server 20 (No), step 505 (S505) is repeated and the object remains in standby mode. For example, when a user enters a room, the query output unit 205 outputs a query output signal to the setting unit 203 in response to a signal output from the server. When the setting unit 203 receives the query output signal, it transitions the object from standby mode to query mode. When the setting unit 203 transitions the object to query mode, it outputs a transition signal to the query output unit 205. This is step 505 (S505).

[0055] If step 505 (S505) has already been executed, step 501 (S501) and step 503 (S503) may be omitted.

[0056] In step 507 (S507), the inquiry output unit 205 outputs an inquiry output signal regarding a predetermined process when it receives a transition signal from the setting unit 203. For example, the inquiry output unit 205 outputs a signal as to whether or not to turn on the lights in the room to the display control unit 201, the setting unit 203, and the operation determination unit 207. This is step 507 (S507).

[0057] In step 509 (S509), when the display control unit 201 receives a signal as to whether or not to turn on the lights in the room, data of a partial area of ​​the virtual space is output to the display unit 105 (FIG. 2(A)). The partial area of ​​the virtual space is displayed on the display unit 105. At this time, as shown in FIG. As shown in (C), if the partial area 310A of the virtual space in which the object 300 is placed and the inquiry output 330-1 are displayed on the display unit 105 (Yes), the information processing device 10 executes step 513 (S513). That is, if the object 300 outputs the inquiry 330-1 to the user, the information processing device 10 executes step 513 (S513). As shown in FIG. 4 (B) or FIG. 4 (C), if a partial area of ​​the virtual space in which no object is placed is displayed on the display unit 105 (No), the information processing device 10 returns to step 505 (S505). That is, if the object 300 does not output the inquiry 330-1 to the user or if the object is not displayed, the information processing device 10 returns to step 505 (S505). The above is step 509 (S509). Note that the user may move the information processing device 10 so that the partial area of ​​the virtual space in which the object is placed is displayed on the display unit 105.

[0058] Next, the user performs input for the object inquiry by moving the information processing device 10. For example, when the display control unit 201 displays an inquiry output about whether to turn on the light on the display unit 105, the detection unit 113 (FIG. 2(A)) detects the movement of the information processing device 10 by the user. This is step 513 (S513).

[0059] When the movement of the information processing device 10 is the first movement, and an input detection signal is output from the movement determination unit 207 to the execution unit 209, a first predetermined process is executed by the execution unit 209 (step 521 (S521)). When the movement of the information processing device 10 is the second movement, and an input non-detection signal is output from the movement determination unit 207 to the execution unit 209, a second predetermined process is executed by the execution unit 209 (step 531 (S531)).

[0060] For example, when the object 300 outputs an inquiry to the user as to whether or not to turn on the light, the user can input "Yes" (first movement) by moving the information processing device 10 up and down as shown in FIG. 5(A). The movement determination unit 207 determines that "Yes" has been input in response to the inquiry as to whether or not to turn on the light, and outputs a signal to turn on the light to the execution unit 209. Furthermore, the execution unit 209 outputs an execution control signal to the server 20, thereby turning on the light in the room. Furthermore, when the object 300 outputs an inquiry to the user as to whether or not to turn on the light, the user can input "No" (second movement) by moving the information processing device 10 left and right as shown in FIG. 5(B). The movement determination unit 207 determines that "No" has been input in response to the inquiry as to whether or not to turn on the light, and outputs a signal to the execution unit 209 not to turn on the light. Furthermore, the execution unit 209 outputs an execution control signal to the server 20, thereby not turning on the light in the room. The above is step 521 (S521) and step 531 (S531). Then, the operation of the information processing device 10 is completed.

[0061] When the operation is completed, the information processing device 10 may return to the standby mode indicated in step 507 (S507) and execute step 509 (S509).

[0062] The input for the inquiry according to the embodiment of the present invention is shown as an example of two-choice input of "yes" and "no", but is not limited to this example. For example, three-choice input of "yes", "no" and "not sure" may be used, or four or more choices may be used. The input for the inquiry according to the embodiment of the present invention can be made to handle complex predetermined processing by increasing the number of choices. Furthermore, the input for the inquiry according to the embodiment of the present invention is shown as an example of "yes" and "no" corresponding to FIG. 5(A) and FIG. 5(B), but is not limited to this example. For example, "yes" and "no" may be corresponding to FIG. 5(B) and FIG. 5(A).

[0063] As an example of the predetermined processing according to an embodiment of the present invention, an example in which a response is made by the movement of the information processing device of the user to an inquiry as to whether or not to turn on the light of an object has been shown. The predetermined processing is not limited to this example. For example, a response may be made by the movement of the information processing device of the user to an inquiry related to the movement of an object.

[0064] As described above, by using an information processing device and program according to one embodiment of the present invention, a user can input a response to an inquiry about an object placed in a virtual space by simple movements of the information processing device.

[0065] In addition, by using the information processing device and the program according to an embodiment of the present invention, the user can input a response by moving the information processing device, which improves the interruption of operations and the cumbersome finger movements that occur in the case of touch operations.

[0066] A processing method when multiple objects are placed in a virtual space will be described with reference to FIG.

[0067] 8 is a diagram showing a virtual space in which a plurality of objects are arranged according to an embodiment of the present invention. Descriptions of configurations that are the same as or similar to those in FIGS. 1 to 7 may be omitted.

[0068] Fig. 8 differs from the diagram shown in Fig. 3 in that a plurality of objects (object 300A, object 300B, object 300C) are arranged in virtual space 305. Fig. 8 also shows an example in which a query mode is set for the objects.

[0069] 8, an object 300A and a query output 330A are placed in a partial area 310A of the virtual space 305. An object 300B and a query output 330B are placed in a partial area 310B of the virtual space 305. An object 300C is placed in a partial area 310C of the virtual space 305. No object is placed in a partial area 310D of the virtual space 305.

[0070] The flowcharts of FIGS. 6 and 7 can also be used when multiple objects (object 300A, object 300B) are placed in virtual space 305.

[0071] For example, regarding a processing method in which a plurality of objects are arranged in a virtual space of the present invention, steps 501 (S501) to 507 are executed based on the flowcharts of FIG. 6 and FIG. 7, and then step 509 (S509) is executed.

[0072] When the partial area 310A of the virtual space 305 in which the object 300A is placed or the partial area 310B of the virtual space 305 in which the object 300B is placed is displayed on the display unit 105 of the information processing terminal (Yes), the user can input to the inquiry through the information processing device 10 in step 513 (S531). On the other hand, when the partial area 310C of the virtual space 305 in which the object 300C is placed is displayed on the display unit 105, the object 300C is in the inquiry mode but has not output an inquiry (No), so the information processing device 10 returns to step 505 (S505). Also, when the partial area 310C of the virtual space 305 in which no object is placed is displayed on the display unit 105 (No), the user does not input to the inquiry, and the information processing device 10 returns to step 505 (S505).

[0073] There are three objects in the inquiry mode, and an inquiry is output, and the user can input a response to the inquiry through the information processing device 10 to the objects displayed on the display unit 105.

[0074] For example, when both partial area 310A of virtual space 305 and partial area 310B of virtual space 305 are displayed on display unit 105, the user inputs a query to one of object 300A or object 300B through information processing device 10, and then inputs a query to the other of object 300A or object 300B through information processing device 10. Furthermore, for example, information processing device 10 may have a function capable of simultaneously inputting queries to a plurality of objects in a query mode.

[0075] For example, each of the multiple objects may be operated by a different user, or may be operated by the same user. The number of displayed objects is not limited to the numbers shown in Figures 3 and 8. The user may appropriately determine the operation of the objects, the number of objects to be displayed, etc.

[0076] As described above, by using the information processing device and program according to an embodiment of the present invention, a user can input a response to an inquiry about a plurality of objects by simply moving the information processing device.

[0077] A processing method according to one embodiment of the present invention will be described with reference to Fig. 9 and Fig. 10. A program according to one embodiment of the present invention includes displaying an image captured by an imaging unit 107 included in an information processing device 10 and an object arranged in a virtual space in a superimposed manner, and responding to an inquiry about the object by moving the information processing device 10.

[0078] Fig. 9 is a flowchart showing a processing method according to an embodiment of the present invention. Fig. 10(A) is a plan view showing a virtual space in which an object is placed according to an embodiment of the present invention. Fig. 10(B) is a plan view showing a captured image according to an embodiment of the present invention. Fig. 10(C) is a plan view showing a virtual space and a captured image superimposed according to an embodiment of the present invention. Descriptions of configurations that are the same as or similar to those in Figs. 1 to 8 may be omitted.

[0079] Step 501 (S501) has already been described using the flowchart shown in Fig. 6, and therefore the description here will be omitted. As described using the flowchart shown in Fig. 6, if step 505 (S505) has already been executed, step 501 (S501) and step 503 (S503) may be omitted.

[0080] In step 504 (S504), for example, the user reads out image data of the object 300 and the virtual space 305, for which one of a plurality of operation modes is set, and image data of the captured image 410, from the server 20. The read image data of the object 300 and the virtual space 305, and the image data of the captured image 420 are input to the display control unit 201 (FIG. 2(B)) via the network 40. Next, the display control unit 201 generates composite image data of a composite image 500 in which image data of a partial area 310 of the virtual space 305 in which the object 300 is arranged and image data of the captured image 410 are superimposed. The display control unit 201 displays the composite image 500 in which the image of the partial area 310 of the virtual space 305 in which the object 300 is arranged and the captured image 410 are superimposed on the display unit 105. This is step 504 (S504). Here, similarly to step 503 (S503), the object is in a state of waiting for an instruction (input) from the user or server 20 (standby mode).

[0081] The steps after step 505 (S505) have already been described using the flowcharts shown in FIGS. 6 and 7, and therefore will not be described here.

[0082] As described above, by using the information processing device and program according to one embodiment of the present invention, even when an image of a partial area of ​​the virtual space in which the object is placed is displayed overlapping with a captured image, the user can input a response to an inquiry about the object by a simple movement of the information processing device. That is, the simple movement of the information processing device is determined from an image obtained by synthesizing an image of a partial area of ​​the virtual space in which the object is placed with the captured image. Therefore, by using the information processing device and program according to one embodiment of the present invention, a response can be input by a simple movement of the information processing device even in response to an inquiry about an object including a captured image, and therefore the discontinuity of operation and the complicated finger movement that occur in the case of a touch operation are improved.

[0083] The processes executed in each of the above-described embodiments of the present invention may be considered as predetermined processes as a whole.

[0084] The above-described embodiments of the present invention can be implemented in any suitable combination as long as they are not mutually inconsistent.

[0085] Even if the effects and advantages brought about by the aspects of each of the above-mentioned embodiments are different from those brought about by the aspects of each of the above-mentioned embodiments, those which are clear from the description in this specification or which can be easily predicted by a person skilled in the art are naturally understood to be brought about by the present invention. [Explanation of symbols]

[0086] 1: information processing system, 10: information processing device, 20: server, 40: network, 100: hardware configuration, 101: communication unit, 103: operation unit, 105: display unit, 105A: display surface, 107: imaging unit, 109: control unit, 111: storage unit, 113: detection unit, 200: functional configuration, 201: display control unit, 203: setting unit, 205: output unit, 207: operation determination unit, 209: execution unit, 300: Object, 300A: object, 300B: object, 300C: object, 305: virtual space, 310: partial area, 310A: partial area, 310B: partial area, 310C: partial area, 310D: partial area, 320B: partial area, 330: output, 330-1: output, 330-2: output, 330A: output, 330B: output, 410: image, 420: image, 500: composite image

Claims

1. displaying at least a part of the virtual space in which the object is placed on a display unit included in the terminal; outputting a query regarding a predetermined process; receiving an input for the inquiry based on a movement of the terminal when at least a part of the object is displayed on the display unit; and executing the predetermined process based on an input in response to the inquiry.

2. A plurality of operation modes are set for the object; and 2. The program according to claim 1, further comprising: setting a predetermined mode in the object; and executing the predetermined process when the query is output from the object.

3. A plurality of the objects are arranged in the virtual space; and one of a plurality of operation modes is set corresponding to each of the plurality of objects; accepting an input in response to the query based on a movement of the terminal; 2. The program according to claim 1, for causing a computer to execute the predetermined process based on an input in response to the inquiry.

4. The program according to any one of claims 1 to 3, for causing a computer to execute the following: the movement of the terminal is determined from an image obtained by synthesizing at least a portion of a virtual space in which the object is placed with an image captured by an imaging unit.

5. The program according to claim 4 , for causing a computer to execute: when the predetermined process is executed based on the movement of the terminal, the image captured by the imaging unit is fixed.

6. The program described in any one of claims 1 to 5, for causing a computer to execute the specified processing, in which the specified processing is executed based on the movement of the terminal when a change amount of a parameter indicating the movement of the terminal is within a specified range.

7. a display control unit that causes the terminal to display at least a part of the virtual space in which the object is placed; an inquiry output unit that outputs an inquiry regarding a predetermined process; An information processing device comprising: an execution unit that accepts input to the query based on movement of the terminal when at least a portion of the object displayed on the terminal is displayed, and executes the specified processing based on the input to the query.

Images