Information providing device and information providing method

Abstract

To efficiently perform virtual object arrangement and display.SOLUTION: A server 10 comprises an arranging unit 133 and a display control unit 131. The arranging unit 133 arranges a virtual object in a virtual space imitating the real world. The display control unit 131 displays, on an editing terminal 30 and a browsing terminal 40, a first image in which the object is arranged in an image of the virtual space. Moreover, the display control unit 131 displays, on a site terminal 20, a second image in which the object is arranged in a captured image of the real world.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information providing device and an information providing method. [Background technology]

[0002] Conventionally, there is known a technique for placing a virtual object in an image of a real-world object, and for displaying the placed virtual object by superimposing it on the image (see, for example, Patent Document 1 and Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-148712 [Patent Document 2] Patent Publication No. 2021-156634 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional techniques have a problem in that the placement and display of virtual objects may not be performed efficiently.

[0005] For example, in the techniques described in Patent Documents 1 and 2, images captured by a camera are required when arranging and displaying virtual objects. Therefore, if it is difficult to obtain captured images, it becomes difficult to arrange virtual objects in desired locations and to display virtual objects that have already been arranged. [Means for solving the problem]

[0006] The information providing device of the embodiment is characterized by having an arrangement unit that arranges virtual objects in a virtual space that mimics the real world, and a display control unit that displays a first image in which the objects are arranged in an image of the virtual space, and a second image in which the objects are arranged in a captured image of the real world. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an overview of an information providing system. [Figure 2] FIG. 2 is a diagram illustrating an overview of the field terminal. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of the field terminal. [Figure 4] FIG. 4 is a diagram showing an example of a screen displayed on the field terminal. [Figure 5] FIG. 5 is a diagram showing an example of a virtual image. [Figure 6] FIG. 6 is a diagram showing an example of a virtual image. [Figure 7] FIG. 7 is a flowchart showing the flow of processing by the server. [Figure 8] FIG. 8 is a flowchart showing the flow of processing by the server. [Figure 9] FIG. 9 is a flowchart showing the flow of processing by the server. DETAILED DESCRIPTION OF THE INVENTION

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An information providing apparatus and an information providing method according to the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.

[0009] [First embodiment] An overview of the information providing system will be described using Fig. 1. Fig. 1 is a diagram illustrating an overview of the information providing system. As shown in Fig. 1, the information providing system 1 has a server 10, a field terminal 20, an editing terminal 30, and a viewing terminal 40. The server 10 is connected to each terminal via a network N so as to be able to communicate data with them.

[0010] The on-site terminal 20, the editing terminal 30, and the viewing terminal 40 each have a different role. However, the roles of each terminal may be realized by a single terminal. That is, the information providing system 1 may have one or more terminals that have all the functions of the on-site terminal 20, the editing terminal 30, and the viewing terminal 40. The on-site terminal 20, the editing terminal 30, and the viewing terminal 40 can be realized by a general personal computer, a smartphone, a tablet, etc.

[0011] Here, Patent Document 1 describes a method of placing an AR object in an image captured by a camera provided in a terminal (distance measurement system) and storing position information of the AR object. The AR object is a virtual 3D object.

[0012] In contrast, the information provision system 1 of this embodiment places an AR object in a virtual space that mimics the real world. At this time, the position in the virtual space where the AR object is placed by the information provision system 1 has a corresponding position in the real world. For example, absolute positions in both the real world and the virtual space can be specified using latitude and longitude. Therefore, the information provision system 1 can reproduce the placement of the AR object in the virtual space in an image captured in the real world.

[0013] Before describing the server 10, the on-site terminal 20 will be described. Fig. 2 is a diagram illustrating an overview of the on-site terminal. As shown in Fig. 2, a user U1 holds the on-site terminal 20 in his / her hand and takes an image using the on-site terminal 20. A building 811, a building 812, a person 813, a tree 814, and a tree 815 exist around the user U1.

[0014] The user U1 can take an image using a camera provided in the on-site terminal 20. The on-site terminal 20 also has the same functions as the distance measurement systems described in Patent Documents 1 and 2.

[0015] 2 is an AR object that has already been placed. The AR object 91 may be placed in the real world via the on-site terminal 20, or may be placed in a virtual space by a method to be described later.

[0016] Fig. 3 is a diagram showing an example of the configuration of a field terminal. As shown in Fig. 3, the field terminal 20 has an information processing device 21 and a distance measuring device 22. For example, the information processing device 21 is a smartphone. The distance measuring device 22 is connected so as to be able to communicate with the field terminal 20 via wired or wireless communication.

[0017] For example, the distance measuring device 22 is an attachment having a mechanism that allows it to be attached to and detached from the information processing device 21. The distance measuring device 22 may be provided with a communication module that complies with a predetermined communication standard and communicate with the information processing device 21 via the communication module.

[0018] The distance measuring device 22 can communicate data with a wide range of devices by using widely used communication standards such as Wi-Fi (registered trademark) and Bluetooth (registered trademark).

[0019] The distance measuring device 22 may be equipped with a battery independent of the information processing device 21. Alternatively, the distance measuring device 22 may be powered by the information processing device 21 via a wired cable.

[0020] The distance measuring device 22 measures the distance to a predetermined surface. At this time, the distance measuring device 22 measures the distance using a distance measuring sensor that uses a laser. Here, the laser has high directivity and linearity, and can measure long distances (for example, 5 to 50 m). For example, the wavelength of the laser used by the distance measuring device 22 may be equal to or greater than the wavelength of visible light and less than the wavelength of infrared light (for example, approximately 600 nm). Furthermore, the distance measuring device 22 can measure the distance to a point in the same direction as the camera provided in the information processing device 21. Furthermore, the distance measuring device 22 transmits the measured distance to the information processing device 21. Note that the distance measuring sensor used in the distance measuring device 22 is not limited to a laser sensor, and any sensor that can grasp 3D space, such as a LiDAR sensor or an infrared sensor, can also be applied.

[0021] The information processing device 21 has an AR function. The AR function may be realized by ARCore (https: / / developers.google.com / ) or ARKit (https: / / developer.apple.com / jp / augmented-reality / arkit / ). For example, the information processing device 21 can place an AR object at a specified location in an image captured by a camera. Here, the information processing device 21 can obtain the accurate distance to a surface, so that the AR object can be placed in close contact with the surface.

[0022] Furthermore, the information processing device 21 can superimpose an AR object on an image captured by a camera and display it. At this time, the information processing device 21 can also display the distance to the surface as a numerical value.

[0023] In this embodiment, the process related to the placement of the AR object is performed by the server 10. The information processing device 21 transmits to the server 10 information required for the placement of the AR object.

[0024] As shown in FIG. 3, the information processing device 21 includes a communication unit 211, a camera 212, an input unit 213, a display unit 214, a positioning unit 215, a storage unit 216, and a control unit 217.

[0025] The communication unit 211 performs wired or wireless data communication with other devices via a network. For example, the communication unit 211 is a network interface card (NIC).

[0026] The camera 212 captures an image of an area including a predetermined point in space.

[0027] The input unit 213 accepts data input from a user. The input unit 213 is, for example, an input device such as a mouse or a keyboard. The display unit 214 is a display device such as a display that displays a screen. The input unit 213 and the display unit 214 may be a touch panel display.

[0028] The positioning unit 215 identifies the position of the information processing device 21 in space. The positioning unit 215 identifies the position using, for example, infrared rays, a beacon signal of a wireless local area network (LAN), or a global navigation satellite system (GNSS) such as GPS, real time kinematic (RTK), or a quasi-zenith satellite system (QZSS). The positioning unit 215 may also have an electronic compass. For example, the electronic compass detects geomagnetism to measure direction.

[0029] The storage unit 216 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM). The storage unit 216 stores an operating system (OS) and programs executed by the information processing device 21. Furthermore, the storage unit 216 stores various types of information used in the execution of the programs.

[0030] The control unit 217 controls the information processing device 21. The control unit 217 is a central processing unit (CPU), a micro processing unit (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like.

[0031] 3, the distance measuring device 22 has a communication unit 221, a control unit 222, and a distance measuring sensor 223. For example, the communication unit 221 is a wireless communication module for communicating data with the information processing device 21. Furthermore, the control unit 222 controls each unit of the distance measuring device 22. For example, the control unit 222 controls the timing at which the distance measuring sensor 223 measures the distance.

[0032] The distance measurement sensor 223 uses a laser to acquire information about the distance from the camera 212 to a reference point. The distance measurement sensor 223 has a light projecting unit 2231 and a light receiving unit 2232. The distance measurement sensor 223 uses a laser to measure the distance by a phase difference detection method, a TOF (Time of Flight) method, or the like.

[0033] Fig. 4 is a diagram showing an example of a screen displayed on the on-site terminal. In the example of Fig. 4, an image (a still image or one frame of a moving image) captured by the camera 212, which includes the existing buildings 811 and 812, is shown in an area 2141.

[0034] As shown in FIG. 4, a setting button 2141a, a distance display field 2141b, a menu control button 2141c, a mode selection button 2141d, and a layout button 2141e are displayed on the left side of the screen.

[0035] The setting button 2141a is a button for setting the AR object to be placed. The distance display field 2141b displays the output value of the distance measuring device 22. For example, the distance display field 2141b displays the distance measured by a laser. The menu control button 2141c is a button for switching between displaying and hiding the option screen on the right. The mode selection button 2141d is a button for selecting a mode.

[0036] The modes include those that change the display manner of the AR object, such as a real display mode and a highlight display mode.

[0037] For example, when the reality display mode is selected, the display unit 214 displays the AR object with a texture that mimics the texture (metal, concrete, resin, etc.) of an actual component.

[0038] On the other hand, when the highlighting mode is selected, the display unit 214 highlights and displays the AR object. For example, the display unit 214 displays the AR object by filling it with a single color or a specific pattern, or by flashing it.

[0039] When the placement button 2141e is operated, the placement of the AR object is determined.

[0040] A separation distance display field 2142a is displayed at the top center of the screen.

[0041] Also displayed on the right side of the screen are a home button 2143a, a movement distance magnification setting field 2143b, an object position coordinate control panel 2143c, an object rotation control panel 2143d, an object drawing control panel 2143e, and a sketch button 2143f.

[0042] The movement distance magnification setting field 2143b sets the movement distance when changing the position coordinates of each object. The object position coordinate control panel 2143c accepts operations for moving an object along each of the three axes in three-dimensional space. The object rotation control panel 2143d accepts operations for rotating an object along a predetermined axis (for example, the vertical axis).

[0043] The object drawing control panel 2143e accepts operations related to the selection and drawing of objects. When the sketch button 2143f is operated, a capture image of the currently displayed screen is acquired and saved.

[0044] As described above, the on-site terminal 20 can arrange an AR object using a method similar to the methods described in Patent Documents 1 and 2.

[0045] Returning to FIG. 1, the server 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0046] The communication unit 11 performs wired or wireless data communication with other devices via a network. For example, the communication unit 11 is a NIC.

[0047] The storage unit 12 is a storage device such as an HDD, SSD, optical disc, RAM, flash memory, or NVSRAM. The storage unit 12 stores an OS and programs executed on the server 10. The storage unit 12 also stores various information used in the execution of the programs. The storage unit 12 stores virtual space information 121 and an object DB 122.

[0048] The virtual space information 121 is information for configuring a virtual space. For example, the virtual space information 121 includes the position, size, shape, etc. of an object (e.g., a building) that exists in the real world. The position in the virtual space information 121 is expressed by latitude and longitude in the real world. Note that the virtual space information 121 may use a known 3D model that imitates the real world (e.g., PLATEAU (https: / / www.mlit.go.jp / plateau / )). The virtual space information 121 may also be generated manually or automatically based on satellite images, stereo camera images, or the like.

[0049] The object DB 122 stores data on placed AR objects. The object DB 122 stores information such as the pattern, texture, size, and shape of the AR object, as well as location information. The location information of the AR object includes latitude, longitude, height, and orientation. The location information may be information specifying the position in the real world and virtual space using a three-axis coordinate system.

[0050] The control unit 13 controls the server 10. The control unit 13 is a CPU, an MPU, an ASIC, an FPGA, etc. The control unit 13 includes a display control unit 131, an identification unit 132, and an arrangement unit 133.

[0051] The display control unit 131 causes a first image (for example, the virtual image 51) in which the AR object 91 is arranged in an image of a virtual space to be displayed on the editing terminal 30 and the viewing terminal 40. The display control unit 131 also causes a second image (for example, the image of the area 2141 in FIG. 4) in which the AR object 91 is arranged in a captured image of the real world to be displayed on the on-site terminal 20.

[0052] The display control unit 131 displays a first image in which the AR object 91 is arranged in an image of a virtual space, and a second image in which the AR object 91 is arranged in a captured image of the real world.

[0053] The display of the first image and the display of the second image may be performed by different display control units. Also, the display of the first image and the display of the second image may be performed by a display control unit provided in the terminal, not in the server 10.

[0054] A display control unit provided in the on-site terminal 20 may cause the first image to be displayed, and a display control unit provided in the editing terminal 30 may cause the second image to be displayed.

[0055] The placement unit 133 places the AR object 91 in a virtual space that imitates the real world.

[0056] The identification unit 132 identifies the location information of the on-site terminal 20 based on the captured image and the sensor values ​​of the on-site terminal 20. For example, the identification unit 132 identifies the location information including the latitude, longitude, height, and orientation of the on-site terminal 20 based on the GPS information, the sensor values ​​of the G sensor, etc. acquired from the on-site terminal 20.

[0057] The placement unit 133 stores the position information of the AR object 91, which is associated with both the real world and the virtual space, in the storage unit 12 (object DB 122). For example, the placement unit 133 stores latitude and longitude as the position information in the storage unit. Note that the position of the virtual space constructed based on the virtual space information 121 is specified by the latitude and longitude.

[0058] When a viewpoint is specified, the display control unit 131 can generate a virtual image seen from the viewpoint in the virtual space. Fig. 5 is a diagram showing an example of the virtual image. The virtual image 51 shown in Fig. 5 is an image of a location in the virtual space that corresponds to the location in the real world shown in Fig. 2.

[0059] Positions in virtual space are expressed using the same three-axis (x, y, z) coordinate system as in the real world. The y-axis is the vertical axis (positive upwards). The x-axis is perpendicular to the y-axis. The z-axis is perpendicular to both the x-axis and y-axis.

[0060] 5, a building object 821 and a building object 822 are displayed on the virtual image 51. The building object 821 and the building object 822 are objects that resemble the building 811 and the building 812 in FIG. 2, respectively.

[0061] An AR object 91 is displayed on the virtual image 51. In Fig. 2, the AR object 91 is displayed on the wall surface of a building object 811. In Fig. 5, the AR object 91 is displayed on the wall surface of a building object 821.

[0062] Note that the building object 821 and the building object 822 are objects that mimic the real world and are not AR objects. Also, the person 813, the tree 814, and the tree 815 in the real world are not reflected in the virtual space. Which objects in the real world are reflected in the virtual space depends on the method for creating the virtual space information 121, etc.

[0063] When the designated viewpoint is changed, the virtual image changes. Fig. 6 is a diagram showing an example of a virtual image. The location corresponding to virtual image 52 in Fig. 6 is near the location corresponding to virtual image 51 in Fig. 5. However, the viewpoints of virtual image 52 in Fig. 6 and virtual image 51 in Fig. 5 are different.

[0064] The processing of the server 10 will be described with reference to Figures 7, 8 and 9. Figures 7, 8 and 9 are flowcharts showing the flow of processing by the server.

[0065] 7 shows a process in which server 10 displays a screen including an AR object on editing terminal 30 and viewing terminal 40. Editing terminal 30 and viewing terminal 40 may display the screen on a flat display or on VR (Virtual Reality) goggles.

[0066] As shown in FIG. 7, first, the server 10 displays a virtual image that imitates the real world based on the virtual space information 121 (step S101).

[0067] Here, the server 10 acquires data of the object (AR object) from the object DB 122 based on the position information corresponding to the virtual image (step S102). For example, the server 10 acquires data of the AR object associated with position information indicating a position within a predetermined range from a position in the virtual space corresponding to the virtual image.

[0068] The server 10 arranges the object on the virtual image based on the acquired object data and displays it in a superimposed manner (step S103).

[0069] For example, in step S101, the server 10 displays an image of the virtual image 51 in Fig. 5 excluding the AR object 91. Then, the server 10 displays the AR object 91 superimposed on the displayed virtual image 51.

[0070] Furthermore, in the editing terminal 30 and the viewing terminal 40, simulations such as distance measurement and collision detection can be performed using the AR object 91 displayed in the virtual image.

[0071] FIG. 8 shows a process when an operation to place an AR object is accepted via the on-site terminal 20 and the editing terminal 30.

[0072] 8, first, the server 10 arranges an object (AR object) on an image in response to an operation and displays it superimposed (step S201). Then, the server 10 stores data of the arranged object, including position information of the object, in the object DB 122 (step S202).

[0073] The image in step S201 may be an image of real space captured by a camera, or may be a virtual image. The operation in step S201 is accepted via a screen UI (User Interface) shown in Fig. 4. The server 10 can accept an operation to place an AR object on the virtual image by providing the editing terminal 30 with a screen displaying the virtual image in the area 2141 in Fig. 4.

[0074] FIG. 9 shows a process in which the server 10 causes the on-site terminal 20 to display a screen including an AR object.

[0075] 9, first, the server 10 acquires the position information of the on-site terminal 20 and the information about the captured image (step S301). Based on the acquired information, the server 10 acquires data of the object (AR object) from the object DB 122 (step S302).

[0076] Then, the server 10 arranges the object on the virtual image based on the acquired data and displays it in a superimposed manner (step S303).

[0077] 7 and 9 may be arranged on a virtual image or on a photographed image of the real world. Furthermore, the server 10 can receive operations such as editing from both the on-site terminal 20 and the editing terminal 30 for AR objects stored in the object DB 122. In other words, the server 10 can synchronize the arranged AR objects between the virtual space and the real space.

[0078] As described above, the server 10 (an example of an information providing device) of the embodiment includes the placement unit 133 and the display control unit 131. The placement unit 133 places a virtual object (e.g., an AR object 91) in a virtual space that imitates the real world. The display control unit 131 causes the editing terminal 30 and the viewing terminal 40 to display a first image (e.g., a virtual image 51) in which the object is placed in an image of the virtual space. The display control unit 131 also causes the on-site terminal 20 to display a second image (e.g., an image of the area 2141 in FIG. 4 ) in which the object is placed in a captured image of the real world.

[0079] In this way, the server 10 can place virtual objects even when no real-world image has been captured. As a result, according to this embodiment, virtual object placement and display can be performed efficiently.

[0080] (Example) An example of the present embodiment will be described below, which is applied to, for example, a construction site.

[0081] The virtual space information 121 is point cloud data of a scanned site (real world). When on-site work begins, a worker who will be conducting a patrol inspection moves to the site with a site terminal 20. A supervisor uses an editing terminal 30 to remotely view the virtual space of the site.

[0082] The server 10 accepts operations such as arranging and changing the AR objects from the on-site terminal 20 and the editing terminal 30, and updates the object DB 122.

[0083] Here, the server 10 acquires the image captured by the on-site terminal 20 and displays a virtual image from the same viewpoint as the captured image on the editing terminal 30. The server 10 may also output the audio collected by the on-site terminal 20 to the editing terminal 30. This allows the supervisor to share the worker's position, line of sight, the image being viewed, and the audio.

[0084] The supervisor uses AR objects to instruct the workers on the work while checking the work status via the editing terminal 30. For example, the supervisor can give instructions to the workers by placing AR objects that output sound, virtual sticky notes (see, for example, Patent Document 1), and CG AR objects (arrows, etc.) that indicate the instructions. The supervisor can also create blueprints for construction, etc. based on the virtual images and provide the created blueprints to the workers.

[0085] Additionally, changes in the real space (switch operations, movement of objects (including objects other than AR objects)) detected by sensors provided in the on-site terminal 20 are notified to the server 10. The server 10 updates the virtual space information 121 in accordance with the notified information. This allows changes in the real world to be quickly reflected in the virtual space.

[0086] Furthermore, the observer can dynamically change the content of the instructions in response to changes reflected in the virtual space.

[0087] In this way, the information provision system 1 enables on-site workers and remote supervisors to work closely together, significantly improving the accuracy and efficiency of work. Furthermore, with the support of a supervisor in the virtual space, the information provision system 1 allows workers to receive expert instructions immediately on-site, speeding up problem resolution. Furthermore, by using sensors and AR devices, real-world movements are accurately reflected in the virtual space, enabling interaction.

[0088] The program executed by the server 10 of this embodiment is provided in a state where it is pre-installed in a ROM or the like. The program executed by the server 10 of this embodiment may be provided by being recorded on a computer-readable recording medium in the form of a file in an installable format or an executable format.

[0089] Furthermore, the program executed by the server 10 of this embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the program executed by the server 10 of this embodiment may be provided or distributed via a network such as the Internet.

[0090] The program executed by the server 10 of this embodiment has a modular structure including the above-mentioned units (display control unit 131, identification unit 132, and placement unit 133), and in actual hardware, the CPU reads and executes the program from the ROM, loading the above-mentioned units onto the main memory, and the display control unit 131, identification unit 132, and placement unit 133 are generated on the main memory.

[0091] Although the embodiments of the present invention have been described, they are presented as examples and are not intended to limit the scope of the invention. The embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0092] 1 Information provision system 10 Servers 11, 211, 221 Communications Department 12, 216 storage section 13, 217, 222 Control section 20 Field terminals 21 Information processing equipment 22 Ranging device 30 Editing terminals 40 Viewing terminals 51, 52 Virtual Images 91 AR Objects 121 Virtual Space Information 122 Object DB 131 Display control unit 132 Specific part 133 Placement section 212 Camera 213 Input section 214 Display section 215 Positioning Unit 223 Distance Sensor 811, 812 Buildings 813 People 814, 815 Trees 821, 822 Building objects 2141 area 2141a Settings button 2141b Distance display field 2141c Menu Control Buttons 2141d Mode selection button 2141e Placement button 2142a Separation distance display field 2143a Home button 2143b Travel distance magnification setting field 2143c Object Position Coordinate Control Panel 2143d Object Rotation Control Panel 2143e Object Drawing Control Panel 2143f Sketch Button 2231 Light projector 2232 Light receiving part

Claims

1. a placement unit that places virtual objects in a virtual space that mimics the real world; a display control unit that displays a first image in which the object is arranged in the image of the virtual space and a second image in which the object is arranged in the photographed image of the real world; An information providing device comprising:

2. The placement unit stores position information of the virtual object, the position information being associated with both the real world and the virtual space, in a storage unit.

2. The information providing device according to claim 1.

3. The placement unit stores the latitude and longitude as the location information in the storage unit.

3. The information providing device according to claim 2.

4. 1. A computer-implemented method for providing information, comprising: an arrangement step of arranging virtual objects in a virtual space that mimics the real world; a first display control step of displaying a first image in which the object is arranged in the image of the virtual space; a second display control step of displaying a second image in which the object is arranged in the captured image of the real world; An information providing method comprising:

Images