Information processing system, information processing method, and program

Abstract

This invention provides an information processing system, information processing method, and program that enable smooth communication with remote locations using digital space. [Solution] An information processing system comprising: a display control unit that displays the digital space on a first display device located in a first location where the model is provided, and on a second display device located in a second location different from the first location, which is a display control unit that receives operations on the second object located in the digital space from a user in the second location, and an operation control unit that controls the operation of the second object and the first object corresponding to the second object based on the operations.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program.

Background Art

[0002] In recent years, technologies related to the digital space have been developed, and various communications using the digital space have been carried out. For example, communication for urban development such as urban planning and community discussions is carried out using a technology called Digital Twin that can represent the environment of the real world in a virtual space and perform simulations. As a technology related to this, Non-Patent Document 1 below discloses a technology in which, by changing the arrangement of models prepared in the real space, the models in the VR (Virtual Reality) space also change correspondingly.

[0003] In addition, by sharing the digital space with a remote location, communication can also be carried out with the remote location. As a technology related to this, Patent Document 1 below discloses a technology that enables a director at a remote location to selectively display a plurality of camera images including HMD (Head Mounted Display) camera images observed by an operator in an MR (Mixed Reality) space where virtual objects are superimposed on a real-shot video. In addition, Patent Document 2 below discloses a technology that enables the transfer of the operation right of a character in a game space to another player.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Non-Patent Documents

[0005] [Non-Patent Document 1] "Technical Verification Report on Experiential Urban Planning Tool Using XR Technology," [online], Ministry of Land, Infrastructure, Transport and Tourism, [Accessed September 9, 2024], Internet<URL:https: / / www.mlit.go.jp / plateau / file / libraries / doc / plateau_tech_doc_0025_ver01.pdf> [Overview of the project] [Problems that the invention aims to solve]

[0006] However, while the technologies described in Patent Documents 1 and 2 above allow for the sharing of viewpoints and operations within a digital space with a remote location, they do not allow for the sharing of model operations as described in Non-Patent Document 1 with a remote location. Therefore, smooth communication with remote locations was sometimes not possible.

[0007] In view of the above-mentioned problems, the object of the present invention is to provide an information processing system, an information processing method, and a program that can facilitate communication with remote locations using digital space. [Means for solving the problem]

[0008] To solve the above-mentioned problems, an information processing system according to one aspect of the present invention is an information processing system comprising: a display control unit that displays the digital space on a first display device located at a first location where the model is provided and on a second display device located at a second location different from the first location; an operation reception unit that receives operations on the second object located in the digital space from a user at the second location; and an operation control unit that controls the operation of the second object and the first object corresponding to the second object based on the operation.

[0009] An information processing method according to one aspect of the present invention is an information processing method performed by a computer that includes: a display control process that causes a real space represented by a model in which a first object capable of autonomous driving in the real world is arranged to be reproduced in a digital space by arranging a second object corresponding to the first object, and the digital space to be displayed on a first display device located in a first location where the model is provided, and on a second display device located in a second location different from the first location; an operation reception process that receives operations on the second object arranged in the digital space from a user in the second location; and an operation control process that controls the operation of the second object and the first object corresponding to the second object based on the operations.

[0010] A program according to one aspect of the present invention is a program for causing a computer to function as a display control means that displays a digital space on a first display device located in a first location where the model is provided, and on a second display device located in a second location different from the first location, which reproduces a real space represented by a model on which a first object capable of autonomous driving in the real world is arranged, by arranging a second object corresponding to the first object in a digital space; an operation receiving means that receives operations on the second object arranged in the digital space from a user in the second location; and an operation control means that controls the operation of the second object and the first object corresponding to the second object based on the operations. [Effects of the Invention]

[0011] According to the present invention, communication with remote locations using digital space can be facilitated smoothly. [Brief explanation of the drawing]

[0012] [Figure 1]This is a diagram showing an example of the configuration of the information processing system according to the present embodiment. [Figure 2] This is a diagram showing an example of a model of the real space according to the present embodiment. [Figure 3] This is a diagram showing an example of an image in the virtual space as viewed from the perspective of an avatar arranged in the virtual space according to the present embodiment. [Figure 4] This is a block diagram showing an example of the functional configuration of the small robot according to the present embodiment. [Figure 5] This is a block diagram showing an example of the functional configuration of the information processing apparatus according to the present embodiment. [Figure 6] This is a block diagram showing an example of the functional configuration of the user terminal according to the present embodiment. [Figure 7] This is a sequence diagram showing an example of the processing flow when the user operates the small robot on-site according to the present embodiment. [Figure 8] This is a sequence diagram showing an example of the processing flow when the user operates the small robot from a remote location according to the present embodiment. [Figure 9] This is a flowchart showing an example of the operation right transfer process according to the present embodiment.

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0014] <1. Configuration of Information Processing System> Referring to FIGS. 1 to 3, the configuration of the information processing system according to the present embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of the information processing system according to the present embodiment.

[0015] The information processing system 1 shown in FIG. 1 reproduces a real space represented by a model in the real world in a digital space, and enables a user located at a first location where the model is provided and a user located at a second location different from the first location to share (communicate) about the real space via the digital space.

[0016] The second location is, for example, a location remote from the first location (remote area). The information processing system 1 is used, for example, to confirm in a digital space at a base B (an example of the second location), which is different from the base A, a real space represented by a model MD (an example of a model) at the base A (an example of the first location). The base A is, for example, the head office of a company, etc. The base B is, for example, a local branch of a company, etc. Note that voice communication may also be performed by connecting the base A and the base B via a voice call.

[0017] Note that the information processing system 1 according to the present embodiment is applicable to, for example, a digital twin (particularly, a digital twin metaverse as a communication platform conscious of real-time interaction) that can represent the environment of the real world in a virtual space and perform simulations, etc., and spatial computing conscious of robots including AI, etc. Hereinafter, an example of the present embodiment will be described by taking as an example the case where, using a digital twin, one floor of a building in the real world (an example of a real space) is represented by a model MD, and one floor of the building is reproduced as a virtual space (an example of a digital space) based on the model MD.

[0018] As shown in FIG. 1, the information processing system 1 includes a small robot 10, an information processing device 20, a display 30, a user terminal 40, and a controller 50.

[0019] (1) Small robot 10 The small robot 10 is an object (an example of a first object) that can autonomously drive in the real world. The small robot 10 performs actions such as moving, emitting light, expanding and contracting, vibrating, and rotating through autonomous driving. In this embodiment, as an example, an example is described in which the small robot 10 is equipped with wheels and a motor and moves autonomously. The small robot 10 is connected to the information processing device 20 via a wired or wireless connection for communication. In this embodiment, as an example, an example in which the small robot 10 is connected to the information processing device 20 via a wireless connection will be described. The communication standard used by the small robot 10 for wireless connection is, for example, BLE (Bluetooth Low Energy, Bluetooth is a registered trademark). In BLE communication, the small robot 10 functions as a peripheral (slave), and the information processing device 20, which is the communication partner, functions as a central (master). As a result, the small robot 10's autonomous drive (movement) is controlled based on instructions received from the information processing device 20.

[0020] The small robots 10 are placed in the model MD. The small robots 10 are used, for example, to represent objects that move in real space in virtual space. Objects that move in real space include, for example, people, animals, robots, and vehicles. One or more small robots 10-1 to 10-n (where n is a natural number) are placed in the model MD. Users at base A can manually operate the small robots 10 placed in the model MD.

[0021] Now, with reference to Figure 2, the model MD of the real space according to this embodiment will be described. Figure 2 is a diagram showing an example of the model MD of the real space according to this embodiment.

[0022] The model MD shown in Figure 2 represents one floor of a building. Two small robots 10-1 and 10-2 are placed in the model MD. Both small robots 10 are positioned assuming the use of the building by its users. Users can freely move each small robot 10 to any position within the model MD. Note that small robot 10-1 is facing the direction indicated by arrow D1, and small robot 10-2 is facing the direction indicated by arrow D2.

[0023] (2) Information processing device 20 The information processing device 20 is composed of, for example, one or more PCs (Personal Computers) or server devices (e.g., cloud servers). In the following, this embodiment will be described using an example where the information processing device 20 is a PC. The information processing device 20 is connected to the small robot 10, the display 30, and the user terminal 40 via wired or wireless connection, enabling communication between them.

[0024] The information processing device 20 is a device that reproduces the real space represented by the model MD in the real world in a virtual space, and performs processing for users at location A where the model MD is located and users at location B to share (communicate about) the real space through the virtual space. When reproducing the real space in the virtual space, the information processing device 20 creates an object corresponding to each small robot 10 placed in the model MD and places it in the virtual space. Such an object is, for example, an avatar (an example of a second object). The information processing device 20 transmits information for displaying the virtual space to the display 30 and the user terminal 40, and displays the virtual space.

[0025] When the small robot 10 in the model MD moves due to the operation of a user at base A, the information processing device 20 receives location information from the small robot 10. Based on the received location information, the information processing device 20 moves the avatar in the virtual space in accordance with the movement of the small robot 10. On the other hand, if an avatar in the virtual space moves due to an operation by a user at base B, the information processing device 20 receives operation information from the user terminal 40. Based on the received operation information, the information processing device 20 moves the small robot 10 in the model MD in accordance with the avatar's movement. Furthermore, if a user at base B attempts to operate the small robot 10, the information processing device 20 controls the transfer of control rights to the small robot 10 to that user, thereby enabling that user to operate the small robot 10 smoothly. Details of the process related to the transfer of control rights to the small robot 10 (hereinafter also referred to as the "control rights transfer process") will be described later.

[0026] (3) Display 30 Display 30 is a display device (an example of a first display device) at base A for displaying a virtual space that reproduces the model MD. Display 30 is connected to the information processing device 20 by a wired or wireless connection. Display 30 displays the virtual space based on the information for displaying the virtual space received from the information processing device 20. If multiple avatars exist in the virtual space, display 30 displays the images in the virtual space from the viewpoint of each avatar together (in a listable format).

[0027] Now, referring to Figure 3, we will describe an image of the virtual space as seen from the viewpoint of an avatar placed in the virtual space according to this embodiment. Figure 3 is a diagram showing an example of an image of the virtual space as seen from the viewpoint of an avatar placed in the virtual space according to this embodiment.

[0028] The display 30 shown in Figure 3 shows a virtual space that recreates the model MD shown in Figure 2. The model MD shown in Figure 2 has small robots 10-1 and 10-2 placed inside it. Therefore, the display 30 shows image G1 of the virtual space as seen from the viewpoint of the avatar AVT1 of small robot 10-1 (image looking in the direction indicated by arrow D1 in Figure 2) and image G2 of the virtual space as seen from the viewpoint of the avatar AVT2 of small robot 10-2 (image looking in the direction indicated by arrow D2 in Figure 2).

[0029] Image G1 displays Avatar AVT2 as seen from Avatar AVT1. Therefore, Image G1 is a first-person perspective image from Avatar AVT1, but it is also a third-person perspective image from Avatar AVT2. Image G2 displays Avatar AVT1 as seen from Avatar AVT2. Therefore, Image G2 is a first-person perspective image from Avatar AVT2, but it is also a third-person perspective image from Avatar AVT1. Thus, when multiple avatar AVTs (small robots 10) exist, the user can view images of the virtual space from the perspective of each avatar AVT, thereby viewing not only first-person perspective images but also third-person perspective images of each avatar.

[0030] (4) User terminal 40 The user terminal 40 is a terminal operated by a user located at site B. The user terminal 40 is, for example, a PC, tablet, or smartphone. The user terminal 40 is connected to the information processing device 20 and the controller 50 via a wired or wireless connection, enabling communication. Based on the information for displaying the virtual space received from the information processing device 20, the user terminal 40 displays the virtual space in the same way as the display 30 described above. In other words, the user terminal 40 is also an example of a second display device.

[0031] When a user at base B operates the controller 50, the user terminal 40 receives the information entered by the user from the controller 50. If the user at base B has not been given control by the information processing device 20, they cannot operate the small robot 10 even if they operate the controller 50. Therefore, the user at site B first operates the controller 50 to perform an operation to request the transfer of control (hereinafter also referred to as "interrupt operation input"). The user terminal 40, which receives the interrupt operation input from the controller 50, sends an alert to the information processing device 20 to request control (hereinafter also referred to as "control alert"). The information processing device 20, upon receiving the control alert, executes the control transfer process. When control rights are transferred through the control rights transfer process, the user at base B operates the controller 50 to perform an operation to move the small robot 10 (hereinafter also referred to as "movement operation input"). The user terminal 40, which receives the movement operation input from the controller 50, transmits operation information indicating the content of the movement operation to the information processing device 20. The information processing device 20, upon receiving the operation information, moves the avatar AVT in the virtual space that is the target of the operation, and also moves the small robot 10 corresponding to the avatar AVT.

[0032] (5) Controller 50 The controller 50 is an input device for a user at base B to input commands to operate a small robot 10 located on model MD at base A. The controller 50 is connected to the user terminal 40 via a wired or wireless connection for communication. The controller 50 inputs interrupt operation input or movement operation input to the user terminal 40 in response to the user's operation at base B.

[0033] <2. Functional Configuration of Small Robots> The configuration of the information processing system 1 according to this embodiment has been described above. Next, an example of the functional configuration of the small robot 10 according to this embodiment will be described with reference to Figure 4. Figure 4 is a block diagram showing an example of the functional configuration of the small robot 10 according to this embodiment. As shown in Figure 4, the small robot 10 includes a communication unit 110, a drive unit 120, a position information acquisition unit 130, a control unit 140, and an alert output unit 150.

[0034] (1) Communications Department 110 The communication unit 110 has the function of sending and receiving various types of information. The communication unit 110 is connected to the information processing device 20 by wired or wireless connection and sends and receives various types of information.

[0035] (2) Drive unit 120 The drive unit 120 has the function of autonomously driving the small robot 10. When the communication unit 110 receives a movement instruction from the information processing device 20, the drive unit 120 drives the motor to autonomously drive the small robot 10. When the small robot 10 has moved to the target position, the drive unit 120 stops the motor and stops the small robot 10.

[0036] (3) Location information acquisition unit 130 The position information acquisition unit 130 has the function of acquiring the position information of the small robot 10. The small robot 10 is equipped with, for example, a reading sensor for acquiring position information on the bottom surface of its housing. In this embodiment, a mat is laid on the floor of the model MD, and position information on the mat is printed on the surface of the mat. The position information acquisition unit 130 acquires the position information of the small robot 10 within the model MD by reading this position information with the reading sensor.

[0037] (4) Control unit 140 The control unit 140 has the function of controlling the overall operation of the small robot 10. The control unit 140 is implemented, for example, by having a microcontroller or similar device, which is included as hardware in the small robot 10, execute a program.

[0038] (5) Alert output unit 150 The alert output unit 150 has a function to output alerts. The alert output unit 150 sends an alert notification (hereinafter also referred to as "operation rights alert notification") to inform a user at base A that a user at base B has requested operation rights. The operation rights alert notification is sent, for example, by vibrating the small robot 10, emitting sound from the small robot 10, or illuminating an LED provided on the small robot 10.

[0039] <3. Functional Configuration of Information Processing Devices> The functional configuration of the small robot 10 according to this embodiment has been described above. Next, an example of the functional configuration of the information processing device 20 according to this embodiment will be described with reference to Figure 5. Figure 5 is a block diagram showing an example of the functional configuration of the information processing device 20 according to this embodiment. As shown in Figure 5, the information processing device 20 includes a communication unit 210, an input unit 220, a storage unit 230, a control unit 240, and an output unit 250.

[0040] (1) Communications Section 210 The communication unit 210 has the function of sending and receiving various types of information. The communication unit 210 is connected to the small robot 10, the display 30, and the user terminal 40 by wired or wireless connection, and sends and receives various types of information.

[0041] (2) Input section 220 The input unit 220 has the function of receiving input from the user. The function of the input unit 220 is realized, for example, by a mouse, keyboard, buttons, touch panel, microphone, etc., provided by the information processing device 20.

[0042] (3) Storage section 230 The storage unit 230 has the function of storing various types of information. The storage unit 230 is composed of storage media provided as hardware by the information processing device 20, such as an HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), RAM (Random Access read / write Memory), ROM (Read Only Memory), or any combination of these storage media.

[0043] (4) Control unit 240 The control unit 240 has the function of controlling the overall operation of the information processing device 20. The control unit 240 is implemented, for example, by causing the CPU (Central Processing Unit) or GPU (Graphics Processing Unit) provided as hardware in the information processing device 20 to execute a program. Alternatively, the control unit 240 may be implemented using software such as Unity® provided by Unity Technologies Inc. or Unreal Engine® provided by Epic Games Inc. As shown in Figure 5, the control unit 240 includes a digital space generation unit 241, a location information acquisition unit 242, a location information management unit 243, an operation reception unit 244, a determination unit 245, an operation rights control unit 246, an operation control unit 247, and a display control unit 248.

[0044] (4-1) Digital space generation unit 241 The digital space generation unit 241 has the function of generating a digital space.

[0045] (4-2) Location information acquisition unit 242 The location information acquisition unit 242 has the function of acquiring location information of the small robot 10 or the avatar AVT. When the small robot 10 is moved by a user at base A, the location information acquisition unit 242 acquires the location information of the small robot 10 after the move. After moving, the small robot 10 reads the location information from the mat and transmits it to the information processing device 20 via the communication unit 110. The location information acquisition unit 242 acquires the location information that the communication unit 210 receives from the small robot 10.

[0046] (4-3) Location information management department 243 The location information management unit 243 has the function of managing the location information of the small robot 10 and the avatar AVT. Based on the location information acquired by the location information acquisition unit 242, the location information management unit 243 updates the location information of the small robot 10 and the location information of the avatar AVT.

[0047] (4-4) Operation reception unit 244 The operation reception unit 244 has the function of receiving operations on the avatar AVT located in the virtual space from a user at base B. For example, when the communication unit 210 receives operation information from the user at base B, the operation reception unit 244 accepts the operation from the user at base B.

[0048] (4-5) Judgment section 245 When an operation is received by the operation reception unit 244, the determination unit 245 determines whether the small robot 10 is in an operational state based on the operation. The determination unit 245 determines whether the small robot 10 is in an operational state based, for example, whether the small robot 10 is in contact with the mat. If it is in contact with the mat, the determination unit 245 determines that the small robot 10 is in an operational state. On the other hand, if it is not in contact with the mat, the determination unit 245 determines that the small robot 10 is not in an operational state. If the small robot 10 is in contact with the mat, for example, it is when the user at base A is not operating the small robot 10. If the small robot 10 is not in contact with the mat, for example, it is when the user at base A is operating the small robot 10.

[0049] (4-6) Control Unit 246 The control unit 246 has the function of controlling the control rights of the small robot 10 based on the determination result of the determination unit 245 regarding the state of the small robot 10. If the control unit 246 determines that the small robot 10 is in an operational state based on the operation to the avatar AVT, it transfers the control rights of the small robot 10 to the user at base B. On the other hand, if the control unit 246 determines that the small robot 10 is not in an operational state based on the operation to the avatar AVT, it does not transfer the control rights of the small robot 10 to the user at base B.

[0050] (4-7) Operation control unit 247 The motion control unit 247 has the function of controlling the movements of the avatar AVT and the corresponding miniature robot 10 based on operations performed on the avatar AVT. The motion control unit 247 causes the avatar AVT to perform an action in response to the operation, and transmits a signal to the corresponding small robot 10 to perform an action that corresponds to the avatar AVT's action. For example, if the avatar AVT is operated by a user at base B, the motion control unit 247 acquires the distance traveled as input by that operation. Based on the acquired distance traveled, the motion control unit 247 calculates the distance traveled by the small robot 10 in the model MD.

[0051] Furthermore, suppose the aforementioned control unit 246 does not transfer control rights to the user at base B. In this case, the motion control unit 247 sends a signal to the small robot 10 corresponding to the avatar AVT that has been requested to be operated by the user at base B, instructing it to send an alert notification to the user at base A regarding control rights.

[0052] (4-8) Display control unit 248 The display control unit 248 has functions to control various displays. The display control unit 248 displays the virtual space on the display 30 at base A where the model MD is installed, and on the user terminal 40 at base B. When the position information of the avatar AVT is updated as the small robot 10 moves, the display control unit 248 moves the position of the avatar AVT in the virtual space and updates the display of the virtual space on the display 30 and the user terminal 40.

[0053] When multiple avatar AVTs are placed in the virtual space, the display control unit 248 displays images of the virtual space from the perspective of each avatar AVT on the display 30 and the user terminal 40.

[0054] (5) Output section 250 The output unit 250 has the function of outputting various types of information. The output unit 250 is composed of output devices provided as hardware by the information processing device 20, such as display devices such as a display device or a touch screen (touch panel), and audio output devices such as a speaker.

[0055] <4. Functional Configuration of User Terminals> The functional configuration of the information processing device 20 according to this embodiment has been described above. Next, an example of the functional configuration of the user terminal 40 according to this embodiment will be described with reference to Figure 6. Figure 6 is a block diagram showing an example of the functional configuration of the user terminal 40 according to this embodiment. As shown in Figure 6, the user terminal 40 includes a communication unit 410, an input unit 420, a storage unit 430, a control unit 440, and an output unit 450.

[0056] (1) Communications Section 410 The communication unit 410 has the function of sending and receiving various types of information. The communication unit 410 is connected to the information processing device 20 and the controller 50 by wired or wireless connection and sends and receives various types of information.

[0057] (2) Input section 420 The input unit 420 has the function of receiving input from the user. The function of the input unit 420 is realized, for example, by a mouse, keyboard, buttons, touch panel, microphone, etc., provided on the user terminal 40.

[0058] (3) Storage section 430 The storage unit 430 has the function of storing various types of information. The storage unit 430 is composed of storage media provided as hardware by the user terminal 40, such as an HDD, SSD, flash memory, EEPROM, RAM, ROM, or any combination of these storage media.

[0059] (4) Control unit 440 The control unit 440 has the function of controlling the overall operation of the user terminal 40. The control unit 440 is implemented, for example, by causing the CPU or GPU provided as hardware in the user terminal 40 to execute a program. As shown in Figure 6, the control unit 440 comprises an operation processing unit 441 and a display processing unit 442.

[0060] (4-1) Operation processing unit 441 The operation processing unit 441 has the function of processing operations. For example, suppose a user operates the controller 50 and inputs an interrupt operation. In this case, the operation processing unit 441 sends an operation right alert to the information processing device 20 via the communication unit 410. Also, suppose a user operates the controller 50 and inputs a movement operation. In this case, the operation processing unit 441 sends operation information to the information processing device 20 via the communication unit 410.

[0061] (4-2) Display processing unit 442 The display processing unit 442 has a function to perform processing related to display. For example, the display processing unit 442 causes the output unit 450 to display the virtual space based on information for displaying the virtual space that the communication unit 410 receives from the information processing device 20.

[0062] (5) Output section 450 The output unit 450 has the function of outputting various types of information. The output unit 450 is composed of output devices provided as hardware by the user terminal 40, such as display devices such as a display device or a touch screen (touch panel), and audio output devices such as a speaker. The output unit 450 displays, for example, a virtual space.

[0063] <5. Processing Flow> The functional configuration of the user terminal 40 according to this embodiment has been described above. Next, the processing flow according to this embodiment will be described with reference to Figures 7 to 9.

[0064] (1) Processing flow when a user operates a small robot on-site Referring to Figure 7, the processing flow when a user operates the small robot 10 on-site will be explained. Figure 7 is a sequence diagram showing an example of the processing flow when a user operates the small robot 10 on-site according to this embodiment.

[0065] As shown in Figure 7, first, the display control unit 248 of the information processing device 20 displays the virtual space on the display 30 at site A where the model MD is installed, and on the user terminal 40 at site B (step S101).

[0066] Next, suppose a user at base A performs a manual movement operation on the small robot 10 (step S102). The moved miniature robot 10 transmits its new position information to the information processing device 20 (step S103). At this time, the position information acquisition unit 130 of the miniature robot 10 acquires the position information by reading the mat at the new position.

[0067] The location information management unit 243 of the information processing device 20 updates the location information of the small robot 10 and the location information of the avatar AVT based on the location information received from the small robot 10 by the communication unit 210 (step S104). Next, the operation control unit 247 of the information processing device 20 moves the position of the avatar AVT in the virtual space based on the updated position information (step S105). Next, the display control unit 248 updates the display of the virtual space on the display 30 and the user terminal 40 with the image of the virtual space as seen from the viewpoint of the avatar AVT after it has moved (step S106).

[0068] (2) Processing flow when a user operates a small robot from a remote location Referring to Figure 8, the processing flow when a user operates the small robot 10 from a remote location will be explained. Figure 8 is a sequence diagram showing an example of the processing flow when a user operates the small robot 10 from a remote location according to this embodiment.

[0069] As shown in Figure 8, first, the controller 50 receives an operation from a user at site B and inputs an interrupt operation to the user terminal 40 (step S201). When the operation processing unit 441 of the user terminal 40 receives an interrupt operation input from the controller 50, it sends an operation rights alert to the information processing device 20 via the communication unit 410 (step S202).

[0070] When the information processing device 20 receives an operation rights alert from the user terminal 40 via the communication unit 210, it executes an operation rights transfer process (step S203). Details of the operation rights transfer process will be described later with reference to Figure 9.

[0071] Next, the controller 50 receives an operation from the user at base B to whom the operation rights have been transferred through the operation rights transfer process, and inputs a movement operation input to the user terminal 40 (step S204). When the operation processing unit 441 of the user terminal 40 receives a movement operation input from the controller 50, it transmits the operation information to the information processing device 20 via the communication unit 410 (step S205).

[0072] Next, the location information management unit 243 of the information processing device 20 updates the location information of the small robot 10 and the location information of the avatar AVT based on the operation information received by the communication unit 210 from the user terminal 40 (step S206). Next, the operation control unit 247 of the information processing device 20 moves the position of the avatar AVT in the virtual space based on the updated position information (step S207). Next, the motion control unit 247 transmits an instruction (signal) to the small robot 10 corresponding to the moved avatar AVT via the communication unit 210, instructing it to move in the same way as the avatar AVT (step S208). Upon receiving a movement instruction from the information processing device 20, the small robot 10 moves according to the instruction (step S209).

[0073] Next, the display control unit 248 of the information processing device 20 updates the display of the virtual space on the display 30 and the user terminal 40 with the image of the virtual space as seen from the viewpoint of the avatar AVT after it has moved (step S210).

[0074] (3) Flow of the process for transferring control The flow of the operation rights transfer process will be explained with reference to Figure 9. Figure 9 is a flowchart showing an example of the operation rights transfer process according to this embodiment.

[0075] As shown in Figure 9, first, the operation rights control unit 246 monitors whether an interrupt operation has been input from a user at site B (step S2031). Specifically, the operation rights control unit 246 checks whether the communication unit 210 has received an operation rights alert from the user terminal 40. If it has been received, it determines that an interrupt operation has been input; if it has not been received, it determines that no interrupt operation has been input.

[0076] If an interrupt operation is input (step S2032 / YES), the process proceeds to step S2033. On the other hand, if no interrupt operation is input (step S2032 / NO), the process repeats from step S2031.

[0077] If the process proceeds to step S2033, the determination unit 245 checks the status of the small robot 10 that is the target of the operation (step S2033). Specifically, the determination unit 245 checks whether the small robot 10 that is the target of the operation is in contact with the mat, and if it is in contact with the mat, it determines that it is operational, and if it is not in contact with the mat, it determines that it is not operational.

[0078] If it is determined that it is not operational (step S2034 / NO), the process proceeds to step S2035. On the other hand, if it is determined that it is operational (step S2034 / YES), the process proceeds to step S2036.

[0079] If the process proceeds to step S2035, the control unit 246 does not transfer control rights to the user at base B, and the motion control unit 247 sends a signal to the small robot 10 corresponding to the avatar AVT that the user at base B has requested to operate, instructing it to send a control rights alert notification (step S2035). Upon receiving the signal, the small robot 10 sends a control rights alert notification. After the notification, the process repeats from step S2033.

[0080] If the process proceeds to step S2036, the operation rights control unit 246 transfers the operation rights to the user at site B (step S2036).

[0081] The processing flow according to this embodiment has been described above. As described above, the information processing system 1 according to this embodiment reproduces a real space represented by a model MD (model) in which an autonomously driven small robot 10 (first object) is placed in the real world by placing an avatar AVT (second object) corresponding to the small robot 10 in a virtual space (digital space). The system includes a display control unit 248 that displays the virtual space on a display 30 (first display device) located at base A (first location) where the model MD is located, and on a user terminal 40 (second display device) located at base B (second location) different from base A; an operation reception unit 244 that receives operations on the avatar AVT placed in the virtual space from a user at base B; and an operation control unit 247 that controls the operation of the avatar AVT and the corresponding small robot 10 based on the operation.

[0082] With this configuration, not only the viewpoint of the avatar AVT placed in the virtual space, but also the operation of the small robot 10 corresponding to the avatar AVT, which is located in the model MD at base A, can be shared with a user at base B, which is in a remote location. Therefore, the information processing system 1 according to this embodiment enables smooth communication with remote locations using digital space.

[0083] <6. Variation> This embodiment has been described above. Next, modifications of the above-described embodiment will be explained. Each modification described below may be applied to the embodiment individually or in combination. Furthermore, each modification may be applied in place of the configuration described in the embodiment, or it may be applied in addition to the configuration described in the embodiment.

[0084] In the embodiment described above, the real space is one floor of a building, the model representing the real space is a model-based digital representation (MD), and the digital space is a virtual space. However, the embodiment is not limited to this example. For example, the real space may be a park, a city, etc. The model representing the real space may be a map, etc. The digital space may be a mixed reality space in which the real space and digital information are superimposed, or a space that records the real world, such as Street View provided by Google.

[0085] Furthermore, in the embodiments described above, the first location was, for example, the company's headquarters, and the second location (remote location) located far from the first location was, for example, a regional branch office of the company. However, the embodiments are not limited to this example. The first location and the second location (remote location) do not necessarily have to be in different buildings. For example, the first location and the second location may be on different floors within the same building, different rooms on the same floor, or different locations within the same room.

[0086] Furthermore, although the above-described embodiment described an example in which the first object is a small, autonomously driven robot, its shape and form are not particularly limited as long as it is autonomously driven.

[0087] Furthermore, in the above-described embodiment, an example was explained in which an external display 30 is used as the display device at site A and the display of the user terminal 40 is used at site B, but the invention is not limited to this example. For example, the display of the information processing device 20 may be used instead of the external display 30 at site A, or an external display may be used instead of the display of the user terminal 40 at site B.

[0088] Furthermore, although the above-described embodiment described an example in which the controller 50 is provided as hardware, the invention is not limited to such an example. For example, the controller 50 may be a digital controller displayed on the user terminal 40's screen.

[0089] Furthermore, in the above-described embodiment, an example was explained in which the position information acquisition unit 130 acquires the position information of the small robot 10 within the model MD by reading the position information printed on the surface of the mat with a reading sensor, but the invention is not limited to this example. For example, the position information acquisition unit 130 may acquire the position information of the small robot 10 by position estimation using technologies such as LiDAR (Light Detection And Ranging) or SLAM (Simultaneous Localization And Mapping).

[0090] Modified examples of this embodiment have been described above. Furthermore, some or all of the functions of the information processing system 1, the small robot 10, the information processing device 20, and the user terminal 40 in the above-described embodiment may be implemented by a computer. In that case, the functions may be implemented by recording a program for implementing these functions on a computer-readable recording medium, loading the program recorded on this recording medium into the computer system, and executing it. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into the computer system. Furthermore, "computer-readable recording media" may include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs over networks such as the Internet or communication lines such as telephone lines, as well as those that hold programs for a certain period of time, such as volatile memory inside computer systems that act as servers or clients in such cases. Furthermore, the above program may be for the purpose of implementing some of the functions described above, or it may be for the purpose of implementing the above functions in combination with a program already recorded in the computer system, or it may be implemented using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0091] Although embodiments of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to those described above, and various design changes can be made without departing from the spirit of this invention. [Explanation of Symbols]

[0092] 1…Information processing system, 10 (10-1 to 10-n)…Small robot, 20…Information processing device, 30…Display, 40…User terminal, 50…Controller, 110…Communication unit, 120…Drive unit, 130…Location information acquisition unit, 140…Control unit, 150…Alert output unit, 210…Communication unit, 220…Input unit, 230…Storage unit, 240…Control unit, 241…Digital space generation unit, 242…Location information acquisition unit, 243…Location information management unit, 244…Operation reception unit, 245…Determination unit, 246…Operation rights control unit, 247…Motion control unit, 248…Display control unit, 250…Output unit, 410…Communication unit, 420…Input unit, 430…Storage unit, 440…Control unit, 441…Operation processing unit, 442…Display processing unit, 450…Output unit

Claims

1. A display control unit that displays the digital space on a first display device located in a second display device located in a second location different from the first location, which is a second display device located in a second location different from the first location, and which reproduces a real space represented by a model in which a first object capable of autonomous driving in the real world is placed in a digital space by placing a second object corresponding to the first object, An operation reception unit that receives operations on the second object located in the digital space from a user at the second location, Based on the above operation, an operation control unit controls the operation of the second object and the first object corresponding to the second object, An information processing system equipped with the following features.

2. When a plurality of the second objects are arranged in the digital space, the display control unit causes the first display device and the second display device to display images of the digital space as seen from the viewpoint of each object. The information processing system according to claim 1.

3. When the aforementioned operation is accepted, a determination unit determines whether the first object is in a state where it can operate based on the aforementioned operation, Based on the determination result of the state, an operation rights control unit controls the operation rights of the first object, The information processing system according to claim 1, further comprising:

4. If the operation rights control unit determines that the first object is in a state where it can be operated based on the operation, it transfers the operation rights of the first object to the user at the second location. The operation control unit causes the second object to perform an operation corresponding to the operation, and transmits a signal to the first object corresponding to the second object to perform an operation corresponding to the operation of the second object. The information processing system according to claim 3.

5. If the operation rights control unit determines that the first object is not in a state where it can be operated based on the operation, it shall not transfer the operation rights of the first object to the user in the second location. The operation control unit transmits a signal to the first object corresponding to the second object for which an operation has been requested by the user at the second location, causing the first object to send an alert notification to the user at the first location. The information processing system according to claim 3 or claim 4, further comprising:

6. A display control process that reproduces a real-world space represented by a model in which a first autonomously driven object is placed in the real world by placing a second object corresponding to the first object in a digital space, and displays the digital space on a first display device located in a first location where the model is provided, and on a second display device located in a second location different from the first location. An operation reception process that receives operations on the second object located in the digital space from a user at the second location, An operation control process that controls the operation of the second object and the first object corresponding to the second object based on the aforementioned operation, A computer-based information processing method that includes [a specific component / function].

7. Computers, A display control means for displaying the digital space on a first display device located in a first location where the model is installed, and on a second display device located in a second location different from the first location, which is a digital display device that reproduces a real space represented by a model in which a first object capable of autonomous driving in the real world is placed, by placing a second object corresponding to the first object in a digital space, and An operation receiving means for receiving operations on the second object located in the digital space from a user at the second location, An operation control means for controlling the operation of the second object and the first object corresponding to the second object based on the aforementioned operation, A program designed to function as such.

Images