Information processing system, information processing method, and program

Abstract

This invention provides an information processing system, information processing method, and program that can ensure consistency between the state of objects existing in the real world and the state of objects existing in the digital world. [Solution] An information processing system comprising: a status management unit that controls changes to status variables that define the operation of both objects based on the state of both objects when the state of one of the objects changes in a real space represented by a model on which a first object is placed, and a digital space that reproduces the real space by placing a second object corresponding to the first object; and an operation control unit that controls the operation of the first object and the second object based on the status variables.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program.

Background Art

[0002] In recent years, technologies related to the digital space have been developed, and various technologies for linking the real space and the digital space have been proposed.

[0003] For example, Patent Document 1 below discloses a technology in which when the state of an object existing in one space changes between an object existing in the real space and an object existing in the virtual space, the state of the object existing in the other space is changed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technology described in Patent Document 1 above, when there is a change in the state of an object existing in one space, the state of the object existing in the other space is not considered, and an instruction for changing the state is unidirectionally transmitted. For this reason, there are cases where the consistency between the state of an object existing in the real space and the state of an object existing in the digital space cannot be ensured.

[0006] In view of the above problems, an object of the present invention is to provide an information processing system, an information processing method, and a program capable of ensuring the consistency between the state of an object existing in the real space and the state of an object existing in the digital space.

Means for Solving the Problems

[0007] To solve the above-mentioned problems, an information processing system according to one aspect of the present invention is an information processing system comprising: a status management unit that controls changes to status variables that define the operation of both objects based on the state of both objects when the state of one of the objects changes in a real space represented by a model on which a first object is placed, and a digital space that reproduces the real space by placing a second object corresponding to the first object; and an operation control unit that controls the operation of the first object and the second object based on the status variables.

[0008] An information processing method according to one aspect of the present invention is an information processing method performed by a computer that includes: a status management process that controls changes to status variables that define the operation of both objects based on the state of both objects when the state of one of the objects changes in a real space represented by a model on which a first object is placed, and a digital space on which a second object corresponding to the first object is placed; and an operation control process that controls the operation of the first object and the second object based on the status variables.

[0009] A program according to one aspect of the present invention is a program that causes a computer to function as a status management means that controls changes to status variables that define the operation of both objects based on the state of both objects when the state of one of the objects changes in a real space represented by a model on which a first object is placed, and a digital space which reproduces the real space by placing a second object corresponding to the first object, and an operation control means that controls the operation of the first object and the second object based on the status variables. [Effects of the Invention]

[0010] According to the present invention, it is possible to ensure consistency between the state of an object existing in the real world and the state of an object existing in the digital world. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of the configuration of the information processing system according to this embodiment. [Figure 2] This figure shows an example of a model of the real space according to this embodiment. [Figure 3] This figure shows an example of an image within the virtual space as seen from the viewpoint of a virtual object placed in the virtual space according to this embodiment. [Figure 4] This is a branch office showing an example of the functional configuration of a real-world object according to this embodiment. [Figure 5] This is a block diagram showing an example of the functional configuration of the information processing device according to this embodiment. [Figure 6] This figure shows an example of object information according to this embodiment. [Figure 7] This flowchart shows an example of the startup process flow according to this embodiment. [Figure 8] This flowchart shows an example of the processing flow when the state of an object changes according to this embodiment. [Figure 9] This figure shows a first embodiment according to this embodiment. [Figure 10] This figure shows a second embodiment according to this embodiment. [Figure 11] This figure shows a third embodiment according to this embodiment. [Figure 12] This figure shows a fourth embodiment according to this embodiment. [Figure 13] This figure shows a fifth embodiment according to this embodiment. [Figure 14] This figure shows a sixth embodiment according to this embodiment. [Modes for carrying out the invention]

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0013] <1. Configuration of Information Processing System> Referring to FIGS. 1 to 3, the configuration of the information processing system according to the present embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of the information processing system according to the present embodiment.

[0014] The information processing system 1 shown in FIG. 1 reproduces a real space represented by a model in the digital space, and executes an operation with consistency between an object arranged in the real space and a corresponding object arranged in the digital space.

[0015] The information processing system 1 reproduces, for example, a real space represented by a model MD (an example of a model) in a virtual space (an example of a digital space). Note that the information processing system 1 according to the present embodiment can be applied to, for example, a digital twin (particularly, a digital twin metaverse as a communication platform conscious of real-time interaction) that can represent the environment of the real world in a virtual space and perform simulations, or spatial computing conscious of robots including AI. Hereinafter, as an example, the present embodiment will be described by taking an example of representing a hamster cage (an example of a real space) in the real world by a model MD using a digital twin and reproducing the hamster cage as a virtual space based on the model MD.

[0016] As shown in FIG. 1, the information processing system 1 includes a real object 10 (a first object), an information processing device 20, and a display 30.

[0017] (1) Real Object 10 Real-world object 10 is an object that exists in the real world. Real-world object 10 is also an object that is reproduced in the virtual space. Real-world object 10 may be, for example, the object itself that exists in the real world, or it may be a robot that corresponds to an object that exists in the real world. Objects that exist in the real world include objects that cannot be controlled by a computer, such as people and animals. Robots are objects that can be controlled by a computer. Real-world object 10 may also be a stationary object that does not move. Any of these objects may be operated or moved manually by the user.

[0018] In the following, this embodiment will be described using an example in which a small robot representing a hamster and a small robot representing a hamster wheel are used as the real-world object 10. The small robots perform actions such as movement, light emission, expansion and contraction, vibration, and rotation through autonomous driving. The hamster real-world object 10 includes, for example, a microcontroller, an actuator, and a tire. Therefore, the hamster real-world object 10 can move (an example of action) by controlling the actuator with the microcontroller and driving the tire. Similarly, the hamster wheel real-world object 10 includes, for example, a microcontroller, an actuator, and a rotatably mounted hamster wheel. Therefore, the hamster wheel real-world object 10 can rotate the hamster wheel (an example of action) by controlling the actuator with the microcontroller and driving the hamster wheel.

[0019] If the real-world object 10 is a communicable object, it is connected to the information processing device 20 via a wired or wireless connection. In this embodiment, as an example, an example in which the real-world object 10 is connected to the information processing device 20 via a wireless connection will be described. The communication standard used by the real-world object 10 for wireless connection is, for example, BLE (Bluetooth Low Energy, Bluetooth is a registered trademark). In BLE communication, the real-world object 10 functions as a peripheral (slave), and the information processing device 20, which is the communication partner, functions as a central (master). As a result, if the real-world object 10 is an object that can be operated by computer control, its operation is controlled based on instructions received from the information processing device 20.

[0020] Real-world objects 10 are placed in model MD. Real-world objects 10 are used, for example, to represent objects that move in real space in virtual space. Objects that move in real space include, for example, people, animals, robots, vehicles, etc. One or more real-world objects 10-1 to 10-n (where n is a natural number) are placed in model MD.

[0021] Now, with reference to Figure 2, the model MD of the real space according to this embodiment will be described. Figure 2 is a diagram showing an example of the model MD of the real space according to this embodiment.

[0022] The model MD shown in Figure 2 represents the interior of a hamster cage. Three real-world objects 10 are placed in the model MD. Real-world objects 10-1 and 10-2 are real-world objects representing hamsters. Real-world objects 10-1 and 10-2 are positioned assuming that hamsters are being kept in a cage. The hamster in real-world object 10-1 is assumed to be facing the direction indicated by arrow D1. Reality object 10-3 is a real-world object representing a hamster wheel. Reality object 10-3 is positioned with the assumption that it will be used by a hamster being kept in a cage.

[0023] (2) Information processing device 20 The information processing device 20 is composed of, for example, one or more PCs (Personal Computers) or server devices (e.g., cloud servers). In the following, this embodiment will be described using an example where the information processing device 20 is a PC. The information processing device 20 is connected to the real object 10 and the display 30 in a communicative manner by a wired or wireless connection.

[0024] The information processing device 20 is a device that reproduces the real space represented by the model MD in the real world in a virtual space, and performs processing to ensure consistent operation between the real objects 10 placed in the real space and the corresponding objects placed in the virtual space. When reproducing the real space in the virtual space, the information processing device 20 creates a corresponding virtual object (second object) for each real object 10 placed in the model MD and places it in the virtual space. The virtual object may be a direct representation of an object, or it may be represented as a character or avatar. The information processing device 20 transmits information for displaying the virtual space to the display 30 and displays the virtual space.

[0025] Assume that a real-world object 10 within the model MD is operated by user input or computer control. In this case, the information processing device 20 obtains (receives) object information from the operated real-world object 10 that indicates information about the real-world object 10. Based on the received object information, the information processing device 20 operates the corresponding virtual object in the virtual space in accordance with the operation of the real-world object 10. On the other hand, suppose a virtual object in the virtual space is operating. In this case, the information processing device 20 obtains object information from the operating virtual object in the virtual space, indicating information about that virtual object. Based on the obtained object information, the information processing device 20 operates the corresponding real object 10 in the model MD in accordance with the operation of the virtual object.

[0026] (3) Display 30 Display 30 is a display device for displaying a virtual space that reproduces the model MD. Display 30 is connected to the information processing device 20 by a wired or wireless connection. Display 30 displays the virtual space based on information for displaying the virtual space received from the information processing device 20. If multiple virtual objects exist in the virtual space, display 30 may display the images of the virtual space from the viewpoint of each virtual object together (in a listable format).

[0027] Now, referring to Figure 3, we will describe an image of the virtual space as seen from the viewpoint of a virtual object placed in the virtual space according to this embodiment. Figure 3 is a diagram showing an example of an image of the virtual space as seen from the viewpoint of a virtual object placed in the virtual space according to this embodiment.

[0028] The display 30 shown in Figure 3 shows a virtual space that recreates the model MD shown in Figure 2. As an example, the image displayed is an image of the virtual space as seen from the viewpoint of virtual object 11-1, which corresponds to real object 10-1, one of the real objects 10-1 to 10-3 placed in the model MD shown in Figure 2 (the image as seen in the direction indicated by arrow D1 in Figure 2). Virtual object 11-2 is a virtual object that corresponds to real object 10-2, and virtual object 11-3 is a virtual object that corresponds to real object 10-3.

[0029] <2. Functional Configuration of Reality Objects> The configuration of the information processing system 1 according to this embodiment has been described above. Next, an example of the functional configuration of the real object 10 according to this embodiment will be described with reference to Figure 4. Figure 4 is a block diagram showing an example of the functional configuration of the real object 10 according to this embodiment. As shown in Figure 4, the real object 10 comprises a communication unit 110, a drive unit 120, a sensor unit 130, and a control unit 140.

[0030] (1) Communications Department 110 The communication unit 110 has the function of sending and receiving various types of information. The communication unit 110 is connected to the information processing device 20 by wired or wireless connection and sends and receives various types of information.

[0031] (2) Drive unit 120 The drive unit 120 has the function of operating the real object 10. When the communication unit 110 receives an operation instruction from the information processing device 20, the drive unit 120 is driven by the control unit 140 and operates the real object 10.

[0032] (3) Sensor unit 130 The sensor unit 130 has the function of acquiring information sensed by the sensor device (hereinafter also referred to as "sensing information"). The real object 10 is equipped with a reading sensor on the bottom surface of its housing for acquiring position information. In this embodiment, a mat is laid on the floor of the model MD, and position information on the mat is printed on the surface of the mat. The sensor unit 130 acquires the position information of the real object 10 within the model MD by reading this position information with the reading sensor. Furthermore, the real-world object 10 may be equipped with sensor devices (e.g., an accelerometer, a gyroscope, a motion sensor, etc.) for acquiring posture information. Furthermore, the real-world object 10 may be equipped with sensor devices (e.g., image sensors, distance sensors, etc.) for acquiring information about the environment surrounding its own device.

[0033] (4) Control unit 140 The control unit 140 has the function of controlling the overall operation of the real object 10. The control unit 140 is implemented, for example, by having a microcontroller or similar device, which is installed as hardware in the real object 10, execute a program. As shown in Figure 4, the control unit 140 includes a state detection unit 141.

[0034] (4-1) State detection unit 141 The state detection unit 141 has the function of detecting the state of the real object 10. For example, the state detection unit 141 detects whether or not there has been a change in the state of the real object 10 based on sensing information acquired by the sensor unit 130. If a change in state is detected, the state detection unit 141 transmits information indicating the content of the change in state (hereinafter also referred to as "state change information") to the information processing device 20. As an example, if it is detected that the real object 10 has moved, the state detection unit 141 transmits object information including state change information indicating that the real object 10 has moved to the information processing device 20.

[0035] <3. Functional Configuration of Information Processing Devices> The functional configuration of the real object 10 according to this embodiment has been described above. Next, an example of the functional configuration of the information processing device 20 according to this embodiment will be described with reference to Figures 5 to 6. Figure 5 is a block diagram showing an example of the functional configuration of the information processing device 20 according to this embodiment. As shown in Figure 5, the information processing device 20 includes a communication unit 210, an input unit 220, a storage unit 230, a control unit 240, and an output unit 250.

[0036] (1) Communications Section 210 The communication unit 210 has the function of sending and receiving various types of information. The communication unit 210 is connected to the real object 10 and the display 30 by wired or wireless connection and sends and receives various types of information.

[0037] (2) Input section 220 The input unit 220 has the function of receiving input from the user. The function of the input unit 220 is realized, for example, by a mouse, keyboard, buttons, touch panel, microphone, etc., provided by the information processing device 20.

[0038] (3) Storage section 230 The storage unit 230 has the function of storing various types of information. The storage unit 230 is composed of storage media provided as hardware by the information processing device 20, such as an HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), RAM (Random Access read / write Memory), ROM (Read Only Memory), or any combination of these storage media.

[0039] (4) Control unit 240 The control unit 240 has the function of controlling the overall operation of the information processing device 20. The control unit 240 is implemented, for example, by causing the CPU (Central Processing Unit) or GPU (Graphics Processing Unit) provided as hardware in the information processing device 20 to execute a program. Alternatively, the control unit 240 may be implemented using software such as Unity® provided by Unity Technologies Inc. or Unreal Engine® provided by Epic Games Inc. As shown in Figure 5, the control unit 240 includes a digital space management unit 241, an object information acquisition unit 242, an object information management unit 243, a status management unit 244, an operation control unit 245, and a display control unit 246.

[0040] (4-1) Digital Space Management Department 241 The Digital Space Management Unit 241 has functions for managing the digital space. For example, the Digital Space Management Unit 241 generates a virtual space that reproduces the model MD, places virtual objects in the virtual space, detects changes in the state of virtual objects, and controls animations in the virtual space.

[0041] (4-2) Object information acquisition unit 242 The object information acquisition unit 242 has the function of acquiring object information. For example, the object information acquisition unit 242 acquires object information of the real object 10 from the real object 10. The object information acquisition unit 242 also acquires object information of virtual objects from the digital space management unit 241.

[0042] (4-3) Object Information Management Unit 243 The object information management unit 243 has the function of managing object information. The object information management unit 243 holds and manages the object information of all objects placed in the real space and virtual space in a control class. Specifically, the object information management unit 243 defines a control class object in the control class space that integrates the corresponding real object 10 in the real space and the virtual object in the virtual space. Furthermore, the object information management unit 243 defines status variables in the control class space to express the macroscopic behavior of all kinds of objects in the real space and virtual space environments that are integrated and managed in an equivalent format. The number of status variables to be defined is not particularly limited. When an individual object's state changes, it changes the corresponding status variable and sends (pushes) the value to the information processing device 20, or receives (pulls) the value of the status variable from the management system and performs the corresponding microscopic behavior.

[0043] The object information management unit 243 registers the object information obtained when each object is registered within the control class. When object information is obtained by the object information acquisition unit 242, the object information management unit 243 updates the corresponding object information within the control class based on the acquired object information. Each object does not communicate directly with its corresponding object in the real or virtual space, but instead passes its object information to the information processing device 20 for management within the control class. This allows the information processing device 20 to refer to the state changes of all objects and send individual messages or instructions to each object.

[0044] Now, with reference to Figure 6, the object information according to this embodiment will be described. Figure 6 is a diagram showing an example of object information according to this embodiment.

[0045] In the example shown in Figure 6, as an example, a hamster real object 10-1, a hamster real object 10-2, and a hamster wheel real object 10-3 are placed in the real space. The object information of these real objects 10 includes, for example, identification information (registration ID), object type, position coordinates, orientation, and state (state change information). Furthermore, the virtual space contains a virtual hamster object 11-1 corresponding to the real object 10-1, a virtual hamster object 11-2 corresponding to the real object 10-2, and a virtual hamster wheel object 11-3 corresponding to the real object 10-3. The object information of these virtual objects 11 includes, for example, an identification ID, object type, position coordinates, orientation, and information indicating the state (state change information). Object-oriented instances are created in the control class space. In the example shown in Figure 6, object OBJ-1 is created corresponding to real object 10-1 and virtual object 11-1, object OBJ-2 corresponds to real object 10-2 and virtual object 11-2, and object OBJ-3 corresponds to real object 10-3 and virtual object 11-3. The object information of these object OBJs includes, for example, an identification ID, object type, serial number, corresponding object in virtual space, coordinates in real space, coordinates in virtual space, orientation, and status (status variable).

[0046] (4-4) Status Management Unit 244 The status management unit 244 has the function of managing the status of objects. For example, suppose the state of one of the corresponding objects changes in the real space where the real object 10 is located and the virtual space where the virtual object 11 corresponding to the real object 10 is located. In this case, the status management unit 244 controls the change of status variables that define the behavior of both objects based on the state of both objects.

[0047] Furthermore, suppose there is a change in the state of an object in one of the spaces. In this case, the status management unit 244 changes the status variable according to the change in state. After the change, the object in the other space is in a state where it can perform actions corresponding to the changed status variable. In this case, the status management unit 244 retains the changed status variable without changing it. On the other hand, suppose an object in the other space is unable to perform an action corresponding to the changed status variable. In this case, the status management unit 244 reverts the changed status variable back to its state before the change.

[0048] Furthermore, suppose there is a change in the state of an object in one space, but the object in the other space is unable to perform the action corresponding to the change in state. In this case, the status management unit 244 retains the status variable without changing it.

[0049] Furthermore, if multiple objects are placed in each space, and multiple objects placed in the same space operate in conjunction, suppose there is a change in the state of an object in one of the spaces. In this case, the status management unit 244 changes the status variable of the object in accordance with the change in state, and also changes the status variables of the linked objects in accordance with the change in the status variable of the object.

[0050] (4-5) Operation control unit 245 The motion control unit 245 has the function of controlling the movement of objects. Based on the status variables after control by the status management unit 244, the motion control unit 245 controls the movement of the real object 10 and the virtual object.

[0051] For example, suppose the state of an object in one space changes, and its status variable is modified. In this case, the operation control unit 245 causes the object in the other space to perform an action corresponding to the modified status variable. However, suppose the object in the other space is in a state where it cannot perform an action, and the modified status variable reverts to the previous status variable. In this case, the operation control unit 245 causes the object in one space to perform an action corresponding to the previous status variable.

[0052] Furthermore, suppose there is a change in the state of an object in one space, but the object in the other space is unable to perform an action corresponding to the change in state, and the status variable remains unchanged. In this case, the operation control unit 245 prevents the object in the other space from performing an action and maintains its current state.

[0053] When multiple objects are placed in each space, and multiple objects placed in the same space operate in conjunction, suppose the state of an object in one space changes, and the status variable of the linked object is changed. In this case, the operation control unit 245 causes the linked objects placed in both spaces to perform an action corresponding to the changed status variable.

[0054] Furthermore, a value indicating whether or not to execute an action can be set in the status variable. Therefore, even if the object in one space and the corresponding object in the other space are of different types, and the status variable indicates that an action should be executed, the action control unit 245 can execute an action appropriate to each type of object.

[0055] (4-6) Display control unit 246 The display control unit 246 has functions to control various displays. The display control unit 246 causes the display 30 to display a virtual space.

[0056] (5) Output section 250 The output unit 250 has the function of outputting various types of information. The output unit 250 is composed of output devices provided as hardware by the information processing device 20, such as display devices such as a display device or a touch screen (touch panel), and audio output devices such as a speaker.

[0057] <4. Processing Flow> The functional configuration of the information processing device 20 according to this embodiment has been described above. Next, the processing flow according to this embodiment will be described with reference to Figures 7 to 8.

[0058] (1) Startup process flow Referring to Figure 7, an example of the startup process flow according to this embodiment will be described. Figure 7 is a flowchart showing an example of the startup process flow according to this embodiment.

[0059] As shown in Figure 7, first, the object information management unit 243 of the information processing device 20 creates an instance of the control class (step S101).

[0060] Next, the object information management unit 243 assigns the pre-registered object information (such as the identification ID) to the instance (step S102).

[0061] Next, the status management unit 244 of the information processing device 20 initializes the status variables (step S103).

[0062] Next, the communication unit 210 of the information processing device 20 establishes communication with the real object 10 (step S104).

[0063] Then, the object information management unit 243 links the real object 10 with the instance (step S105).

[0064] (2) Processing flow when an object's state changes Referring to Figure 8, an example of the processing flow when the state of an object changes according to this embodiment will be described. Figure 8 is a flowchart of an example of the processing flow when the state of an object changes according to this embodiment. In the flowchart shown in Figure 8, the first space may be either real space or virtual space, and the second space may also be either real space or virtual space. However, if the first space is real space, the second space will be virtual space. If the first space is virtual space, the second space will be real space.

[0065] As shown in Figure 8, if there is a change in the state of the object in the first space (step S201 / YES), the process proceeds to step S202. On the other hand, if there is no change in the state of the object in the first space (step S201 / NO), the process repeats from step S201.

[0066] If the process proceeds to step S202, and the object information acquisition unit 242 acquires object information from the object in the first space (step S202 / YES), the process proceeds to step S203. On the other hand, if the object information acquisition unit 242 does not acquire object information from the object in the first space (step S202 / NO), the process repeats from step S201.

[0067] If the process proceeds to step S203, the status management unit 244 of the information processing device 20 identifies the object to be subject to status management based on the object information acquired by the object information acquisition unit 242 (step S203).

[0068] Next, the status management unit 244 refers to the instance of the identified object (step S204).

[0069] Next, the status management unit 244 changes the status variables based on the object information acquired by the object information acquisition unit 242 (step S205).

[0070] Next, the operation control unit 245 of the information processing device 20 issues an operation instruction to the corresponding object in the second space (step S206).

[0071] Next, if the object information acquisition unit 242 acquires object information from the object that has been instructed to operate by the operation control unit 245 (step S207 / YES), the process proceeds to step S208. For example, if the object that has been instructed to operate is in a state where it cannot operate, object information is transmitted from that object. Therefore, the object information acquisition unit 242 acquires this object information. On the other hand, if the object information acquisition unit 242 does not acquire object information from the object that has been instructed to perform an operation by the operation control unit 245 (step S207 / NO), the process is repeated from step S201. For example, if the object that has been instructed to perform an operation is able to perform the operation successfully, no object information is transmitted from that object. Therefore, the object information acquisition unit 242 will not acquire object information.

[0072] If the process proceeds to step S208, the status management unit 244 identifies the object to be subject to status management based on the object information acquired by the object information acquisition unit 242 (step S208).

[0073] Next, the status management unit 244 refers to the instance of the identified object (step S209).

[0074] Next, the status management unit 244 changes the status variable based on the object information acquired by the object information acquisition unit 242 (step S210). For example, the status management unit 244 returns the value of the status variable to the value it was before it was changed in step S205.

[0075] Next, the motion control unit 245 issues an operation instruction to the corresponding object in the first space (step S211). After the operation command is given, the process repeats from step S201.

[0076] The processing flow according to this embodiment has been described above. As described above, the information processing system 1 according to this embodiment includes a status management unit 244 that controls changes to status variables that define the operation of both objects based on the state of both objects when the state of one of the objects changes in a real space represented by a model MD on which a real object 10 (first object) is placed, and a virtual space (digital space) that reproduces the real space by placing a virtual object 11 corresponding to the real object 10, and an operation control unit 245 that controls the operation of the real object 10 and the virtual object based on the status variables.

[0077] With this configuration, if there is a change in the state of an object in one space, instructions are conveyed after considering the state of the object in the other space. Therefore, it is possible to ensure consistency between the state of objects in the real space and the state of objects in the digital space, and to make each object perform the appropriate action. Therefore, the information processing system 1 according to this embodiment makes it possible to ensure consistency between the state of objects existing in the real world and the state of objects existing in the digital world.

[0078] <5. Examples> This embodiment has been described above. Next, an example of the above-described embodiment will be explained with reference to Figures 9 to 14. Note that "move," shown as an example in Figures 9 to 14, is one of the status variables and is defined as "move." Therefore, "move=ON" means "move," and "move=OFF" means "do not move." In the following embodiment, the initial value of the status variable is assumed to be "OFF."

[0079] (1) First embodiment A first embodiment will be described with reference to Figure 9. Figure 9 is a diagram showing the first embodiment according to this embodiment.

[0080] As shown in Figure 9, first, let's assume that the hamster real object 10-1 moves in real space (step S11). The real object 10-1 transmits a state change (move=ON) to the control class space (step S12). The information processing device 20 changes the status variable of object OBJ-1, which corresponds to the real object 10-1, to "move=ON" (step S13). The information processing device 20 issues an operation instruction to the virtual object 11-1 in the virtual space corresponding to the real object 10-1, based on the changed status variable (move=ON) (step S14). The virtual object 11-1 performs actions similar to those of the real object 10-1 based on operation instructions from the information processing device 20 (step S15).

[0081] (2) Second embodiment A second embodiment will be described with reference to Figure 10. Figure 10 is a diagram showing a second embodiment according to this embodiment.

[0082] As shown in Figure 10, first, let's assume that the virtual object 11-1 of the hamster moves in the virtual space (step S21). The virtual object 11-1 transmits a state change (move=ON) to the control class space (step S22). The information processing device 20 changes the status variable of object OBJ-1, which corresponds to virtual object 11-1, to "move=ON" (step S23). The information processing device 20 issues an operation instruction to the real object 10-1 in the real space corresponding to the virtual object 11-1, based on the changed status variable (move=ON) (step S24). The real object 10-1 performs actions similar to those of the virtual object 11-1 based on operation instructions from the information processing device 20 (step S25).

[0083] (3) Third embodiment A third embodiment will be described with reference to Figure 11. Figure 11 is a diagram showing the third embodiment according to this embodiment.

[0084] As shown in Figure 11, first, let's assume that the virtual object 11-1 of the hamster moves in the virtual space (step S31). The virtual object 11-1 transmits a state change (move=ON) to the control class space (step S32). The information processing device 20 changes the status variable of object OBJ-1, which corresponds to virtual object 11-1, to "move=ON" (step S33). Based on the changed status variable (move=ON), the information processing device 20 issues an operation instruction to the real object 10-1 in the real space that corresponds to the virtual object 11-1 (step S34). The real object 10-1 received an operation command from the information processing device 20, but was unable to move due to a malfunction (step S35). Therefore, the real object 10-1 transmits a state change (move=OFF) to the control class space (step S36). The information processing device 20 changes the status variable of object OBJ-1, which corresponds to the real object 10-1, to "move=OFF" (step S37). The information processing device 20 issues an operation instruction to the virtual object 11-1 in the virtual space corresponding to the real object 10-1, based on the changed status variable (move=OFF) (step S38). Based on the operation instructions from the information processing device 20, the virtual object 11-1 stops in accordance with the immobile real object 10-1 (step S39).

[0085] (4) Fourth embodiment A fourth embodiment will be described with reference to Figure 12. Figure 12 is a diagram showing the fourth embodiment according to this embodiment.

[0086] As shown in Figure 12, first, assume that the virtual object 11-1 of the hamster moves in the virtual space (step S41). The action at this time is flying, which is an action that is impossible for the real object 10-1 corresponding to the virtual object 11. For this reason, the virtual object 11 does not communicate the change in state to the information processing device 20 (i.e., it does not transmit object information). The information processing device 20 maintains the status variable of the corresponding object OBJ-1 as "move=OFF" because a state change has occurred in the virtual object 11-1 but has not been communicated (step S42). The information processing device 20 issues an operation instruction to the real object 10-1 in the real space that corresponds to the virtual object 11-1, based on the status variable (move=OFF) (step S43). The real object 10-1 received an operation instruction from the information processing device 20, but remains motionless because there is no change in the status variable (step S44).

[0087] (5) Fifth embodiment A fifth embodiment will be described with reference to Figure 13. Figure 13 is a diagram showing a fifth embodiment according to this embodiment. In the fifth embodiment, an example will be described in which multiple objects placed in the same space operate in conjunction with each other. Note that "Approach" shown as an example in Figure 13 is one of the status variables and is defined as "approach to another object". Therefore, "Approach = ON" means that "approaching another object" and "Approach = OFF" means that "not approaching another object".

[0088] As shown in Figure 13, first, let's assume that the virtual object 11-1 of the hamster moves in the virtual space (step S51). Let's assume that this movement causes the virtual object 11-1 to approach the virtual object 11-3 of the exercise wheel. Therefore, the virtual object 11-1 transmits a state change (approach = ON) to the control class space (step S52). The information processing device 20 changes the status variable of object OBJ-3, which corresponds to virtual object 11-3 that virtual object 11-1 has approached, to "approach = ON" via object OBJ-1, which corresponds to virtual object 11-1 (step S53). Based on the changed status variable (approach = ON), the information processing device 20 issues operation instructions to the real object 10-3 in the real space and the virtual object 11-3 in the virtual space that correspond to object OBJ-3 (step S54). The real object 10-3 and the virtual object 11-3 perform movement based on operation instructions from the information processing device 20 (step S55).

[0089] (6) Sixth embodiment Referring to Figure 14, a sixth embodiment will be described. Figure 14 is a diagram showing the sixth embodiment according to this embodiment. The status variable defines the type of action (such as move), but it does not define the details of the action, such as how it will move specifically. The details of the action can be defined for each object, for example. Therefore, the action of an object when "move=ON" can be made different for each object. That is, when "move=ON", it is possible to set corresponding objects in the real space and the virtual space to perform different actions. An embodiment of this case will be described.

[0090] As shown in Figure 14, first, assume that the virtual object 11-1 of the hamster moves in the virtual space (step S61). As a result of this movement, assume that the virtual object 11-1 approaches the virtual object 11-3 of the exercise wheel. Therefore, the virtual object 11-1 transmits a state change (approach = ON) to the control class space (step S62). The information processing device 20 changes the status variable of object OBJ-3, which corresponds to virtual object 11-3 that virtual object 11-1 has approached, to "approach = ON" via object OBJ-1, which corresponds to virtual object 11-1 (step S63). Based on the changed status variable (approach = ON), the information processing device 20 issues operation instructions to the real object 10-4 (fan) in the real space and the virtual object 11-3 (hamster wheel) in the virtual space, which correspond to object OBJ-3 (step S64). Based on the operation instructions from the information processing device 20, the real object 10-4 performs the operation to move the fan, and the virtual object 11-3 performs the operation to rotate the hamster wheel (step S65).

[0091] The embodiments of this model have been described above. Based on the first and second embodiments, the information processing system 1 according to this embodiment can link objects in real space and objects in virtual space in real time and bidirectionally. Furthermore, in the third and fourth embodiments, if the states of corresponding objects in the real and virtual spaces diverge, each object moves so that the status variable of the control class matches the highest priority state, or the actual state is not pushed to the control class. Therefore, the information processing system 1 according to this embodiment can ensure consistency in the state of each object. Furthermore, in the fifth embodiment, each object can cooperate with other objects without directly interfering with them by essentially entrusting the determination of its own behavior to the information processing device 20 during the processing it performs. In addition, the flexibility of the system can be increased by representing the interactions of various objects with calculations of a finite number of variables. Furthermore, as shown in the sixth embodiment, the information processing system 1 according to this embodiment does not require any changes to the system's core algorithm even when an unknown object (for example, a fan) is newly incorporated into the environment.

[0092] <6. Variation> Next, modifications of the above-described embodiment will be explained. Each modification described below may be applied to the embodiment individually or in combination. Furthermore, each modification may be applied in place of the configuration described in the embodiment, or it may be applied in addition to the configuration described in the embodiment.

[0093] In the embodiment described above, the real space is a hamster cage, the model representing the real space is a model MD, and the digital space is a virtual space. However, the embodiment is not limited to this example. For example, the real space may be a building, a park, a city, etc. The model representing the real space may be a map, etc. The digital space may be a mixed reality space in which the real space and digital information are superimposed, or a space that records the real world, such as Street View provided by Google.

[0094] Furthermore, although the above-described embodiment mentions an example in which an external display 30 is used as the display device, the invention is not limited to this example. For example, the display of the information processing device 20 may be used instead of the external display 30.

[0095] Furthermore, although the above-described embodiment describes an example in which the information processing device 20 is connected to a communicable real object 10 via a wired or wireless connection and receives object information from the real object 10, the invention is not limited to such an example. For example, the information processing device 20 may acquire information sensed by various sensor devices about the real object 10 as object information from various sensor devices.

[0096] Furthermore, in the above-described embodiment, an example was explained in which the sensor unit 130 acquires the position information of the real object 10 within the model MD by reading position information printed on the surface of the mat with a reading sensor, but the invention is not limited to this example. For example, the sensor unit 130 may acquire the position information of the real object 10 by position estimation using techniques such as LiDAR (Light Detection And Ranging) or SLAM (Simultaneous Localization And Mapping).

[0097] Modified examples of this embodiment have been described above. Furthermore, some or all of the functions of the information processing system 1, the real object 10, and the information processing device 20 in the above-described embodiment may be implemented by a computer. In that case, the functions may be implemented by recording a program for implementing these functions on a computer-readable recording medium, and then having the computer system read and execute the program recorded on this recording medium. In this context, "computer system" includes hardware such as the operating system and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Additionally, "computer-readable recording media" may also include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over networks such as the Internet or communication lines such as telephone lines, as well as those that hold programs for a fixed period of time, such as volatile memory within computer systems that act as servers or clients in such cases. Furthermore, the above program may be for the purpose of implementing some of the functions described above, or it may be for the purpose of implementing the above functions in combination with a program already recorded in the computer system, or it may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0098] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to those described above, and various design changes can be made without departing from the spirit of this invention. [Explanation of Symbols]

[0099] 1...Information processing system, 10 (10-1 to 10-n)...Real object, 11 (11-1 to 11-3)...Virtual object, 20...Information processing device, 30...Display, 110...Communication unit, 120...Drive unit, 130...Sensor unit, 140...Control unit, 141...State detection unit, 210...Communication unit, 220...Input unit, 230...Storage unit, 240...Control unit, 241...Digital space management unit, 242...Object information acquisition unit, 243...Object information management unit, 244...Status management unit, 245...Operation control unit, 246...Display control unit, 250...Output unit

Claims

1. In a real space represented by a model on which a first object is placed, and in a digital space that reproduces the real space by placing a second object corresponding to the first object, when the state of one of the objects changes, a status management unit controls the change of status variables that define the behavior of both objects based on the states of both objects. An operation control unit that controls the operation of the first object and the second object based on the status variable, An information processing system equipped with the following features.

2. The status management unit, when there is a change in the state of an object in one space, changes the status variable according to the change in state, and when the object in the other space is in a state where it can perform an action according to the changed status variable, it retains the changed status variable without changing it. The operation control unit causes the object in the other space to perform an action corresponding to the changed status variable. The information processing system according to claim 1.

3. If the status management unit finds that an object in the other space is unable to perform an action corresponding to the changed status variable, it will revert the changed status variable back to its state before the change. The operation control unit causes the object in one space to perform an action corresponding to the status variable before the change. The information processing system according to claim 2.

4. The status management unit, when there is a change in the state of an object in one space, but the object in the other space is unable to perform an action corresponding to the change in state, retains the status variable without changing it. The operation control unit causes the object in the other space to maintain its current state. The information processing system according to claim 1.

5. When multiple objects are placed in each space, and multiple objects placed in the same space operate in conjunction with each other, The status management unit, when there is a change in the state of an object in one of the spaces, changes the status variable of the object in accordance with the change in state, and also changes the status variable of linked objects in accordance with the change in the status variable of the object. The operation control unit causes the interconnected objects located in both spaces to perform actions corresponding to the changed status variables. The information processing system according to claim 1.

6. The aforementioned status variable is set to a value that indicates whether or not to perform the action. The operation control unit, when the object in one space and the corresponding object in the other space are of different types, and the status variable indicates that the operation should be performed, causes each of the objects to perform the operation corresponding to its type. The information processing system according to claim 5.

7. In a real space represented by a model on which a first object is placed, and in a digital space that reproduces the real space by placing a second object corresponding to the first object, when the state of one of the objects changes, a status management process controls the change of status variables that define the behavior of both objects based on the states of both objects. An operation control process that controls the operation of the first object and the second object based on the status variable, A computer-based information processing method that includes [a specific component / function].

8. Computers, In a real space represented by a model on which a first object is placed, and in a digital space that reproduces the real space by placing a second object corresponding to the first object, when the state of one of the objects changes, status management means control changes to status variables that define the behavior of both objects based on the states of both objects. An operation control means that controls the operation of the first object and the second object based on the status variable, A program designed to function as such.

Images