Information processing system

Abstract

To provide an information processing system capable of reducing a time required for the arrangement confirmation of a camera on a simulator.SOLUTION: In an information processing system 1, a calibration part 111 matches a coordinate of an XR (cross reality) space with a coordinate of a simulation space. An XR space construction part 112 arranges an object including an object of a camera on the XR space. A simulation space construction part 211 arranges the object on the simulation space so that a position and an attitude on the XR space correspond to a position and an attitude on the simulation space. A view point display control part 114 displays a viewpoint image in the simulation space viewed from the object of the camera on a display part.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system. [Background technology]

[0002] Patent Document 1 discloses a technology that generates a virtual viewpoint image by generating and rendering a three-dimensional model based on camera images, and performs a camera installation simulation to determine whether each camera, after installation, can capture images suitable for generating a three-dimensional model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-118042 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technology disclosed in Patent Document 1 requires that a camera be actually positioned for the simulation, and there is a problem in that it takes time to check the position of the camera on the simulator.

[0005] In consideration of such problems, the present disclosure aims to provide an information processing system that can reduce the time required to check the placement of cameras on a simulator. [Means for solving the problem]

[0006] The information processing system of the present disclosure includes a calibration unit that aligns coordinates of XR (cross reality) space with coordinates of a simulation space, an XR space construction unit that arranges objects including a camera object in the XR space, a simulation space construction unit that arranges the objects in the simulation space so that their positions and orientations in the XR space correspond to their positions and orientations in the simulation space, and a viewpoint display control unit that displays a viewpoint image in the simulation space as seen from the camera object on a display unit.

[0007] Furthermore, the information processing system of the present disclosure further includes a registration unit that registers the simulation space in which one or more of the objects are arranged.

[0008] With the above-described configuration, the information processing system of the present disclosure does not need to actually place a camera for the simulation, and can reduce the time required to check the placement of the camera on the simulator.

[0009] The information processing system of the present disclosure further includes a placement assistance unit that places an object placed in the simulation space in the XR space so that the position and orientation in the simulation space correspond to the position and orientation in the XR space, displays an image of the XR space in which the object is placed on the display unit, and when a user detects an actual object placed in real space while looking at the position and orientation of the object displayed on the display unit, estimates the orientation and position of the actual object in the XR space and provides assistance to the user based on the difference between the orientation and position of the object in the XR space and the orientation and position of the actual object in the XR space.

[0010] With the above-described configuration, the information processing system of the present disclosure can easily arrange the actual object in the local environment assumed by the simulator after the simulation. Effect of the Invention

[0011] The present disclosure makes it possible to provide an information processing system that can reduce the time required to check the placement of cameras on a simulator. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing an example of a configuration of an information processing system according to a first embodiment. [Diagram 2] 5 is a flowchart showing an example of a layout simulation operation in the information processing system according to the first embodiment. [Diagram 3] FIG. 2 is a diagram showing an example of a screen displayed on an XR device or a simulation space registered in the information processing system according to the first embodiment. [Figure 4] 5 is a flowchart showing an example of a placement support operation in the information processing system according to the first embodiment. [Diagram 5] FIG. 11 is a block diagram showing an example of a configuration of an information processing system according to a second embodiment. [Figure 6] 13 is a flowchart showing an example of a layout simulation operation in the information processing system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and duplicated descriptions will be omitted as necessary for clarity of explanation.

[0014] (First embodiment) First, the configuration of an information processing system 1 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of the information processing system 1 according to the first embodiment.

[0015] As shown in Fig. 1, the information processing system 1 is a system that simulates the placement of cameras and other devices in a local environment. The local environment is, for example, an environment in which an annotated composite image is created from images captured by a camera. The information processing system 1 is also a system that provides support for a user in placing real objects such as cameras in the local environment.

[0016] The information processing system 1 includes an XR (cross reality) device 10 and a simulator device 20. The XR device 10 is, for example, a head mounted display (HMD, also called goggles) that can be worn by a user, and is a device that constructs an XR space by superimposing information such as objects on a real space using XR technology, and displays the constructed XR space. XR technology is a general term for technologies such as AR (Augmented Reality), VR (Virtual Reality), MR (Mixed Reality), and SR (Substitutional Reality).

[0017] The XR device 10 includes a control unit 11, an operation input unit 12, a display unit 13, a communication unit 14, a shooting unit 15, and a storage unit 16. The control unit 11 controls the XR device 10. The operation input unit 12 is, for example, a touch panel, and receives operation input from the user. The display unit 13 is, for example, a transmissive or non-transmissive display, and displays various information to the user. The communication unit 14 is, for example, a communication interface, and communicates with the simulator device 20 wirelessly or via a wire. The shooting unit 15 is, for example, a camera, and shoots the real space around the XR device 10. The angle of view of the shooting unit 15 corresponds to the field of view of the user wearing the XR device 10. The storage unit 16 is, for example, a storage, and stores various information.

[0018] The simulator device 20 is, for example, a PC (Personal Computer) and is a device that constructs a simulation space. The simulation space of the simulator device 20 reproduces to some extent the background of the actual environment in which a camera and the like are placed.

[0019] The simulator device 20 includes a control unit 21, a storage unit 22, a display unit 23, and a communication unit 24. The control unit 21 controls the simulator device 20. The storage unit 22 is, for example, a storage, and stores various information. The display unit 23 is, for example, a transmissive or non-transmissive display, and displays various information to the user. The communication unit 24 is, for example, a communication interface, and communicates with the XR device 10 wirelessly or via a wire.

[0020] The control unit 11 of the XR device 10 includes a calibration unit 111, an XR space construction unit 112, a display control unit 113, a viewpoint display control unit 114, a registration unit 115, and a placement support unit 116.

[0021] The calibration unit 111 matches the coordinates of the XR space constructed by the XR device 10 with the coordinates of the simulation space constructed by the simulator device 20. For example, the calibration unit 111 matches the coordinates by using markers or object position matching. The XR space construction unit 112 constructs an XR space based on the captured image of the real space by the imaging unit 15 and the user's operation of the operation input unit 12. The XR space construction unit 112 places an object selected by the user in the XR space. The objects include a camera, an object that is the subject of the camera, such as a workpiece or a dolly, and a robot. In addition, the XR space construction unit 112 supplies meta information of the object (such as the name of the object, its position in the XR space, and its posture in the XR space) to the simulator device 20 via the communication unit 14.

[0022] The display control unit 113 causes the display unit 13 to display an image of the XR space constructed by the XR space construction unit 112. Based on a user's operation on the operation input unit 12, the viewpoint display control unit 114 causes the display unit 13 to display a viewpoint image in the simulation space as seen from a predetermined camera object arranged in the simulation space. The registration unit 115 registers the constructed simulation space based on the user's operation of the operation input unit 12. The registration unit 115 also registers a viewpoint image in the simulation space seen from a predetermined camera object based on the user's operation of the operation input unit 12.

[0023] The placement support unit 116 places the object placed in the simulation space in the XR space such that the position and orientation in the simulation space correspond to the position and orientation in the XR space, and causes the display unit 13 to display an image of the XR space in which the object is placed. Then, the placement support unit 116 detects an actual object placed in the real space by the user while viewing the position and orientation of the object displayed on the display unit 13, and then estimates the orientation and position of the actual object in the XR space. The placement support unit 116 provides the user with object placement support based on the difference between the orientation and position of the object in the XR space and the orientation and position of the actual object in the XR space.

[0024] The control unit 21 of the simulator device 20 includes a simulation space construction unit 211 . The simulation space construction unit 211 acquires meta information of objects from the XR device 10 via the communication unit 24. The simulation space construction unit 211 constructs a simulation space based on the acquired meta information of objects. Specifically, the simulation space construction unit 211 places the objects in the simulation space such that the position and orientation in the XR space correspond to the position and orientation in the simulation space.

[0025] Next, an operation of simulating the placement of cameras and the like assuming a local environment in the information processing system 1 according to the first embodiment (hereinafter, placement simulation operation) will be described with reference to Fig. 2 and Fig. 3. Fig. 2 is a flowchart showing an example of the placement simulation operation in the information processing system 1 according to the first embodiment. Fig. 3 is a diagram showing an example of an image displayed or a simulation space registered on the XR device 10 of the information processing system 1 according to the first embodiment.

[0026] The following operation is premised on the assumption that the user wears the XR device 10 and activates the XR device 10 and the simulator device 20 in an environment in which a layout simulation operation is to be performed.

[0027] As shown in FIG. 2, first, in step S101, in the information processing system 1, the XR apparatus 10 starts communication with the simulator apparatus 20. Next, in step S102, the calibration unit 111 of the XR device 10 aligns the coordinates between the XR space constructed by the XR device 10 and the simulation space constructed by the simulator device 20.

[0028] Next, the XR space construction unit 112 displays a panel on the display unit 13 from which the user can select an object. The objects include a camera, an object that is a subject of the camera, such as a workpiece or a dolly, and a robot. The user operates the panel to select an object. In step S103, the XR space constructor 112 selects an object in accordance with an operation of the operation input unit 12 by the user.

[0029] Next, in step S104, the XR space construction unit 112 reads the 3D data of the object selected by the user from the storage unit 16, and displays an image (shown in FIG. 3A) in which the 3D data of the object is operably superimposed on the XR space on the display unit 13. FIG. 3A shows an image in which the 3D data of the object displayed on the display unit 13 of the information processing system 1 according to the first embodiment is operably superimposed on the XR space. In FIG. 3A, 3D data X1 of the camera object is displayed operably as the 3D data of the object. The user can adjust the position, orientation, etc. of the 3D data of the object in the XR space by manipulating the 3D data of the object displayed on the display unit 13 with a finger.

[0030] Next, in step S105, the XR space constructing unit 112 places the 3D data of the object in the XR space in accordance with an operation of the operation input unit 12 by the user. Next, in step S106, the XR space construction unit 112 supplies meta information of the 3D data of the object arranged in the XR space to the simulator device 20 via the communication unit 14. The meta information includes the name of the object, its position in the XR space, and its attitude (rotation angle) in the XR space. Note that the meta information may include internal parameters such as the camera resolution, FoV, focal length, and sensor center, and the type such as pinhole or fisheye.

[0031] Next, in step S107, the simulation space construction unit 211 of the simulator device 20 reads, based on the meta-information, 3D data of the object from the storage unit 22. Specifically, the simulation space construction unit 211 reads, from the storage unit 22, 3D data of the object corresponding to the name of the object included in the meta-information.

[0032] Next, in step S108, the simulation space construction unit 211 arranges the 3D data of the loaded object in the simulation space based on the meta-information. Specifically, the simulation space construction unit 211 arranges the 3D data of the loaded object in the simulation space so as to correspond to the position, orientation, etc. in the XR space included in the meta-information.

[0033] When placing another object in the simulation space, the information processing system 1 may perform the processes of steps S103 to S108 again.

[0034] By the above-mentioned processing, in the information processing system 1, the user can easily place an object such as a camera in the simulation space.

[0035] Next, the user presses the Preview button on the screen shown in Fig. 3A displayed on the display unit 13 of the XR device 10. At this time, the camera object placed in the XR space and the simulation space in the above-mentioned process is selected. Note that the user may select another camera object.

[0036] Next, in step S109, the viewpoint display control unit 114 displays a viewpoint image (shown in FIG. 3B) in the simulation space seen from the object of the camera on the display unit 13 in response to the user pressing the Preview button on the operation input unit 12. FIG. 3B shows a viewpoint image in the simulation space seen from the object of the camera displayed on the display unit 13 of the information processing system 1 according to the first embodiment. Specifically, the viewpoint display control unit 114 supplies a Preview instruction to the simulator device 20 via the communication unit 14 in response to the user pressing the Preview button on the operation input unit 12. The simulation space construction unit 211 of the simulator device 20 generates a viewpoint image in the simulation space seen from the object of the camera based on the Preview instruction. The simulation space construction unit 211 supplies the generated viewpoint image to the XR device 10 via the communication unit 24. The viewpoint display control unit 114 of the XR device 10 displays the viewpoint image on the display unit 13.

[0037] Through the above-described processing, in the information processing system 1, the user can determine whether the viewpoint image of the arranged camera is suitable for use in generating a composite image.

[0038] Next, the user presses the Save button on the screen shown in FIG. 3A that is displayed on the display unit 13 of the XR device 10. Next, in step S110, the registration unit 115 registers the simulation space in which the camera object is placed, in response to the user pressing the Save button of the operation input unit 12. Fig. 3C shows the simulation space registered by the registration unit 115 of the information processing system 1 according to the first embodiment. In Fig. 3C, 3D data Y1 of the camera object is placed in the registered simulation space.

[0039] By the above-mentioned process, the information processing system 1 allows the user to register the constructed simulation space.

[0040] Next, an operation of supporting a user in placing an actual object such as a camera in a local environment in the information processing system 1 according to the first embodiment (hereinafter, placement support operation) will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the placement support operation in the information processing system 1 according to the first embodiment.

[0041] In the following operations, it is assumed that the user wears the XR device 10 in the local environment and activates the XR device 10 and the simulator device 20. In the following operations, a camera is placed as a real object in the local environment. It is also preferable to use AR technology as the XR technology.

[0042] First, as shown in FIG. 4, in step S201, the calibration unit 111 of the XR device 10 aligns the coordinates between the XR space created by the XR device 10 and the simulation space created by the simulator device 20.

[0043] Next, in step S202, the placement support unit 116 places the object placed in the simulation space in the XR space so that the object has a corresponding position and orientation, and displays an image of the XR space in which the camera object is placed on the display unit 13. The displayed image is an image of the real space on which the CAD of the camera object is superimposed by AR.

[0044] Next, while viewing the position and orientation of the camera object displayed on display unit 13, the user positions the actual camera so that it is superimposed on the camera object. Next, in step S203, placement assistance section 116 detects, from the image captured by imaging section 15, a real camera that the user has placed while viewing the object displayed on display section 13 so as to be superimposed on the object. Next, in step S204, the placement support unit 116 estimates the position and orientation of the actual camera in the XR space from the image captured by the imaging unit 15. Next, in step S205, when the difference in position and orientation in the XR space between the camera object and the actual camera is equal to or smaller than a threshold, the placement assistance unit 116 changes color or emits light to the camera object displayed on the display unit 13. Note that the placement assistance unit 116 is not limited to changing color or emitting light, and it is sufficient if the camera object displayed on the display unit 13 is made noticeable.

[0045] By the above-described process, in the information processing system 1, the user can receive placement support and thereby reduce the time required to place the actual camera.

[0046] As described above, the information processing system 1 does not require an actual camera to be placed for the simulation, and can reduce the time required to check the placement of the camera on the simulator. Furthermore, the information processing system 1 can easily arrange the actual object in the local environment assumed by the simulator after the simulation.

[0047] Furthermore, when generating an annotated composite image, the camera is positioned and parameters are set in the simulator, assuming the on-site environment. After that, in the on-site environment, a real camera is positioned according to the position in the simulator to check for consistency, and the angle of view, interference with work flow, wiring, etc. are checked. In the comparative example of the information processing system 1, it took 43.5 hours (equivalent to one week) to perform all of these tasks. However, in the information processing system 1 of the present disclosure, it can be completed in about one day.

[0048] Specifically, in the comparative example of the information processing system 1, when simulating the placement of cameras and the like in the local environment, (1) operation confirmation (work posture, cart position, robot, human movement): 4 hours, (2) investigation and confirmation of possible installation range: 2 hours, (3) placement of work in the simulator device: 5 hours, (4) consideration of camera requirements (selection, position, angle of view): 4 hours, (5) work plan for on-site consistency verification: 1.5 hours are performed. In addition, in the comparative example of the information processing system 1, when actually placing cameras in the local environment, (6) work plan creation: 4 hours, (7) meeting: 2 hours, (8) work preparation: 1 hour, (9) installation and wiring work: 1 hour, (10) preparation of transport cart and work: 0.5 hours, (11) angle of view adjustment and confirmation: 1 hour, (12) meeting on installation position as a premise for operation: 1 hour are performed. Therefore, in the comparative example of the information processing system 1, if all of these tasks were performed, it would take 43.5 hours (equivalent to one week). On the other hand, in the information processing system 1, the tasks (1) to (5) can be performed in parallel. In the comparative example of the information processing system 1, the tasks (9) to (11) need to be repeated for each camera installation candidate, and if there are eight camera installation candidates, they are repeated eight times. However, in the information processing system 1, they only need to be performed once. In the comparative example of the information processing system 1, when checking the consistency with the actual object, it is not possible to check by synchronizing and aligning with the simulator side on the spot, and adjustment takes time. However, in the information processing system 1, this is not necessary. In the comparative example of the information processing system 1, if the camera installation position is high or there is an installation candidate in a place where a safety work plan is required, incidental work such as document creation and joint meetings occurs each time. However, in the information processing system 1, this is not necessary. Therefore, in the information processing system 1, what would take 43.5 hours (equivalent to one week) can be completed in about one day.

[0049] Second embodiment First, the configuration of an information processing system 2 according to the second embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the configuration of an information processing system 2 according to the second embodiment.

[0050] The information processing system 2 differs from the information processing system 1 according to the first embodiment in that the construction of a simulation space is performed by an XR device 30 instead of by a simulator device 20.

[0051] The control unit 11 of the XR device 30 includes the configuration of a simulation space construction unit 117 in addition to the configuration of the XR device 10 according to the first embodiment. The simulation space construction unit 117 constructs a simulation space based on meta information of objects placed in the XR space by the XR space construction unit 112. Specifically, the simulation space construction unit 117 places the objects in the simulation space so that their positions and orientations in the XR space correspond to their positions and orientations in the simulation space.

[0052] Next, an object placement simulation operation in the information processing system 2 according to the second embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of a placement simulation operation in the information processing system 2 according to the second embodiment.

[0053] As shown in FIG. 6, in step S301, the calibration unit 111 of the XR device 30 of the information processing system 2 aligns coordinates between the XR space and the simulation space constructed by the XR device 10.

[0054] Next, the XR space construction unit 112 causes a panel from which the user can select an object to be displayed on the display unit 13. The user operates the panel to select an object. In step S302, the XR space constructor 112 selects an object in accordance with an operation of the operation input unit 12 by the user.

[0055] Next, in step S303, the XR space construction unit 112 reads the 3D data of the object selected by the user from the storage unit 16, and displays an image (shown in FIG. 3A) in which the 3D data of the object is superimposed on the XR space in a manipulable manner on the display unit 13. The user can adjust the position, orientation, etc. of the 3D data of the object in the XR space by manipulating the 3D data of the object displayed on the display unit 13 with their fingers.

[0056] Next, in step S304, the XR space constructing unit 112 places the 3D data of the object in the XR space in accordance with an operation of the operation input unit 12 by the user. Next, in step S305, the XR space constructor 112 stores in the storage unit 16 meta information of the 3D data of the object placed in the XR space. Next, in step S306, the simulation space construction unit 211 places the 3D data of the read object in the simulation space based on the meta-information.

[0057] Next, in step S307, the XR space constructing unit 112 copies to the simulator device 20 the 3D data of the object placed in the XR space and meta information of the 3D data of the object.

[0058] The information processing system 2 may be configured to perform the above-described processes of steps S109 and S110 after the process of step S306.

[0059] By the above-described processing, the information processing system 2 according to the second embodiment can perform a simulation of object placement even in an environment in which communication between the XR device 10 and the simulator device 20 is not possible, in addition to the effects of the first embodiment.

[0060] Each of the configurations of the XR device 10, the simulator device 20, and the XR device 30 in the above-mentioned embodiments is configured by hardware or software, or both, and may be configured by one piece of hardware or software, or may be configured by multiple pieces of hardware or software.

[0061] Specifically, each of the XR device 10, the simulator device 20, and the XR device 30 in the above-mentioned embodiment is configured by a computer having a processor and a memory. The processor may be, for example, a microprocessor, a micro processing unit (MPU), or a central processing unit (CPU). The processor may include multiple processors. The memory is configured by a combination of a volatile memory and a non-volatile memory. The memory may include a storage located away from the processor. In this case, the processor may access the memory via an I / O interface not shown. The processor executes one or more programs including a group of instructions for making the computer execute the algorithm described using the drawings.

[0062] The program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0063] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0064] 1,2 Information processing system, 10 XR (cross reality) device, 11 control unit, 12 operation input unit, 13 display unit, 14 communication unit, 15 shooting unit, 16 memory unit, 20 simulator device, 21 control unit, 22 memory unit, 23 display unit, 24 communication unit, 30 XR device, 111 calibration unit, 112 XR space construction unit, 113 display control unit, 114 viewpoint display control unit, 115 registration unit, 116 placement support unit, 117, 211 simulation space construction unit

Claims

1. A calibration unit that aligns coordinates of an XR (cross reality) space with coordinates of a simulation space; an XR space construction unit that arranges objects including a camera object in the XR space; a simulation space construction unit that places the object in the simulation space so that a position and orientation in the XR space corresponds to a position and orientation in the simulation space; a viewpoint display control unit that displays a viewpoint image in a simulation space seen from an object of the camera on a display unit. Information processing system.

2. The simulation space in which one or more of the objects are arranged is further provided with a registration unit. The information processing system according to claim 1 .

3. placing an object placed in the simulation space in the XR space such that its position and orientation in the simulation space correspond to its position and orientation in the XR space, and displaying an image of the XR space in which the object is placed on the display unit; When a user detects an actual object placed in a real space while viewing the position and orientation of the object displayed on the display unit, the orientation and position of the actual object in the XR space are estimated; The apparatus further includes a placement support unit that provides support to the user based on a difference between a posture and a position of the object in the XR space and a posture and a position of the real object in the XR space.

3. The information processing system according to claim 1 or 2.

Images