Image processing device, image processing system device, image processing method, program, and recording medium

Abstract

To provide an image processing apparatus configured to easily set a position of a virtual image to be superposed on a site image.SOLUTION: An image processing apparatus 10 includes: a 3D model generation unit 12 which generates a 3D model representing a configuration in a specific area; a presentation unit 14 which presents the 3D model and a virtual image to an administrator; a coordinate information management unit 15 which sets a reference marker and coordinate information of the virtual image on the basis of, a position in the 3D model designated by the administrator, and manages the virtual image in association with the coordinate information of the virtual image; a site terminal information acquisition unit 16 which acquires position information of a site terminal based on the reference marker and direction information of the site terminal; and an image information output unit 17 which refers to the 3D model, on the basis of, the position information and the direction information of the site terminal, and outputs to the site terminal, when a position corresponding to the coordinate information of the virtual image exists in a site image, a synthesized image obtained by superimposing the virtual image on the position.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an image processing system, an image processing method, a program, and a recording medium. [Background technology]

[0002] BACKGROUND ART Techniques have been reported in which a virtual image is superimposed on a captured image using AR (Augmented Reality) technology (for example, Patent Document 1, etc.). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-006582 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is a demand for a technique that allows for easier setting of the position of a virtual image to be superimposed on a captured image.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device, an image processing system, an image processing method, a program, and a recording medium that make it possible to more easily set the position of a virtual image to be superimposed on a captured image. [Means for solving the problem]

[0006] In order to achieve the above object, the image processing device of the present invention comprises: The system includes an image acquisition unit, a 3D model generation unit, a virtual image generation unit, a presentation unit, a coordinate information management unit, a local terminal information acquisition unit, and an image information output unit, the image acquisition unit acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera, the 3D model generation unit generates a 3D model showing a configuration within the specific area based on the three-dimensional image information; the virtual image generation unit generates a virtual image relating to the specific area; the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area, the presentation unit presents the 3D model and the virtual image to an administrator; the coordinate information management unit sets coordinate information of the reference marker and the virtual image based on a position in the 3D model specified by the administrator, and manages the virtual image and the coordinate information of the virtual image in association with each other; The image acquisition unit further acquires a local captured image captured within the specific area by a local terminal, the local terminal information acquisition unit acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker; The image information output unit is a device that, based on the position information and orientation information of the local terminal, refers to the 3D model to determine whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, outputs a composite image with the virtual image superimposed at that position to the local terminal.

[0007] The image processing method of the present invention comprises: The method includes an image acquisition step, a 3D model generation step, a virtual image generation step, a presentation step, a coordinate information management step, a local terminal information acquisition step, and an image information output step. The image acquisition step acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera, the 3D model generation step generates a 3D model showing a configuration within the specific area based on the three-dimensional image information; the virtual image generating step generates a virtual image relating to the specific area; the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area, the presenting step presents the 3D model and the virtual image to an administrator; The coordinate information management step sets coordinate information of a reference marker and the virtual image based on a position in the 3D model specified by the administrator, and manages the virtual image and the coordinate information of the virtual image in association with each other; The image acquisition step further includes acquiring a local captured image of the specific area captured by a local terminal; the local terminal information acquisition step acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker; The image information output process is a method in which, based on the position information and orientation information of the local terminal, the 3D model is referenced to determine whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, a composite image with the virtual image superimposed on that position is output to the local terminal. [Effects of the Invention]

[0008] According to the present invention, it is possible to more easily set the position of a virtual image to be superimposed on a captured image, thereby enabling an administrator to set the position where the virtual image is to be displayed without, for example, going to a specific area where the captured image is captured. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image processing apparatus according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of the image processing apparatus according to the first embodiment. [Figure 3] FIG. 3 is a flowchart showing an example of processing in the image processing apparatus of the first embodiment. [Figure 4] FIG. 4 is a block diagram showing an example of the configuration of an image processing system apparatus according to the second embodiment. [Figure 5] FIG. 5 is a block diagram illustrating an example of the configuration of a local terminal according to the second embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of processing in the local terminal according to the second embodiment. [Figure 7] FIG. 7 is a block diagram illustrating an example of the configuration of the administrator terminal according to the second embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of processing in the administrator terminal according to the second embodiment. [Figure 9] Figure 9(A) is a schematic diagram showing an example of a 3D model of a specific area in which coordinate information for each virtual image is set, and Figure 9(B) is a schematic diagram showing an example of a top view of the 3D model. [Figure 10] FIG. 10 is a schematic diagram showing an example in which a composite image output from an image processing device to a local terminal is displayed on the display of the local terminal. DETAILED DESCRIPTION OF THE INVENTION

[0010] In the image processing device of the present invention, for example, The presentation unit may present to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to coordinate information of the virtual image within an image included in the three-dimensional image information.

[0011] The image processing device of the present invention includes, for example, It also includes a liaison department. The communication intermediation unit may mediate communication between the administrator and the user having the local terminal.

[0012] The image processing system of the present invention comprises: The image processing device of the present invention and a local terminal are included, the image processing device and the on-site terminal are capable of communicating with each other via a communication network; In the image processing device, the image acquisition unit acquires a local captured image captured by the local terminal within the specific area; The image information output unit is a device that outputs the composite image to the local terminal.

[0013] In the image processing method of the present invention, for example, The presenting step may be a mode in which a preview composite image in which the virtual image is superimposed at a position corresponding to coordinate information of the virtual image within an image included in the three-dimensional image information is presented to the administrator.

[0014] The image processing method of the present invention includes, for example, Further, it includes a liaison intermediation step, The communication intermediation step may be configured to intermediate communication between the administrator and the user having the local terminal.

[0015] The program of the present invention is a program for causing a computer to execute the steps of the method of the present invention as a procedure.

[0016] The recording medium of the present invention is a computer-readable recording medium on which the program of the present invention is recorded.

[0017] Next, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiments. In the following drawings, the same parts are denoted by the same reference numerals. Furthermore, the descriptions of the embodiments can be mutually incorporated unless otherwise specified, and the configurations of the embodiments can be combined unless otherwise specified.

[0018] [Embodiment 1] Fig. 1 is a block diagram showing an example of the configuration of an image processing device 10 according to this embodiment. As shown in Fig. 1, this device 10 includes an image acquisition unit 11, a 3D model generation unit 12, a virtual image generation unit 13, a presentation unit 14, a coordinate information management unit 15, a local terminal information acquisition unit 16, and an image information output unit 17. Furthermore, this device 10 may further include, for example, a communication intermediation unit 18 as an optional component.

[0019] The device 10 may be, for example, a single device including the above-described units, or a device in which the units can be connected via a communication network. The device 10 can also be connected to an external device (described later) via the communication network. The communication network is not particularly limited and any known network can be used, for example, a wired or wireless network. Examples of the communication network include the Internet, the World Wide Web (WWW), a telephone line, a Local Area Network (LAN), a Storage Area Network (SAN), a Delay Tolerant Networking (DTN), a Low Power Wide Area Network (LPWA), and a Local 5G (L5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), Local 5G, and LPWA. Examples of the wireless communication include direct communication between devices (Ad Hoc communication), infrastructure communication, and indirect communication via an access point. The device 10 may be incorporated into a server as a system. The device 10 may be, for example, a personal computer (PC, for example, a desktop or notebook type) on which the program of the present invention is installed, a smartphone, a tablet terminal, etc. Furthermore, the device 10 may be in the form of cloud computing or edge computing, for example, in which at least one of the units is located on a server and the other units are located on a terminal.

[0020] 2 shows an example of a block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a central processing unit (CPU, GPU, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, a display device 106, a communication device 107, an imaging device 108, and an audio input / output device 109. The components of the device 10 are connected to each other via the bus 103 and their respective interfaces (I / F).

[0021] The central processing unit 101 is responsible for overall control of the device 10. In the device 10, the central processing unit 101 executes, for example, the program of the present invention and other programs, and also reads and writes various types of information. Specifically, the central processing unit 101 functions as, for example, an image acquisition unit 11, a 3D model generation unit 12, a virtual image generation unit 13, a presentation unit 14, a coordinate information management unit 15, a local terminal information acquisition unit 16, an image information output unit 17, and a communication intermediation unit 18.

[0022] The bus 103 can also be connected to, for example, an external device. Examples of the external device include an external storage device (such as an external database), a printer, an external input device, an external display device, and an external imaging device. The device 10 can be connected to an external network (the communication line network) by, for example, a communication device 107 connected to the bus 103, and can also be connected to other devices via the external network.

[0023] The memory 102 may be, for example, a main memory (primary storage device). When the central processing unit 101 performs processing, the memory 102 reads various operating programs, such as the program of the present invention, stored in the storage device 104 (described later), and the central processing unit 101 receives data from the memory 102 and executes the programs. The main memory may be, for example, a RAM (random access memory). Alternatively, the memory 102 may be, for example, a ROM (read only memory).

[0024] The storage device 104 is also referred to as an auxiliary storage device, for example, in contrast to the main memory (primary storage device). As described above, the storage device 104 stores an operating program including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive that reads and writes data from and to the recording medium. The recording medium is not particularly limited and may be, for example, an internal or external type, such as a hard disk (HD), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, or memory card. The storage device 104 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD) in which the recording medium and drive are integrated.

[0025] In the present device 10, the memory 102 and the storage device 104 can also store various information such as log information, information acquired from an external database (not shown) or an external device, information generated by the present device 10, and information used when the present device 10 executes processing. Note that at least a portion of the information may be stored, for example, in an external server other than the memory 102 and the storage device 104, or may be stored in a distributed manner across multiple terminals using blockchain technology or the like.

[0026] The device 10 may further include, for example, an input device 105, a display device 106, an imaging device 108, and an audio input / output device 109. The input device 105 is, for example, a touch panel, a keyboard, a mouse, etc. The display device 106 is, for example, an LED display, a liquid crystal display, etc. The imaging device 108 is, for example, a 2D camera, a 3D camera, an infrared camera, etc. The audio input / output device 109 is, for example, a microphone, a speaker, etc.

[0027] Next, an example of the image processing method of this embodiment will be described with reference to the flowchart of Fig. 3. The image processing method of this embodiment is implemented as follows, for example, using the image processing device 10 of Fig. 1. Note that the image processing method of this embodiment is not limited to use of the image processing device 10 of Fig. 1.

[0028] First, the image acquisition unit 11 acquires three-dimensional image information of a specific area captured by a 3D camera (S1). The 3D camera refers to a camera that can capture not only vertical and horizontal information but also depth (depth). The 3D camera is, for example, an external device. The specific area is not particularly limited as long as it is indoors. Examples of indoor areas include exhibition halls, event venues, department stores, and schools such as universities. The three-dimensional image information includes, for example, an image and point cloud data corresponding to the image. The image may be, for example, a still image or a video. The image may also include, for example, multiple panoramic images, which can be stitched together to form a full-surround panoramic image. The acquisition is performed, for example, via the communication network.

[0029] Next, the 3D model generation unit 12 generates a 3D model representing the configuration within the specific area based on the three-dimensional image information (S2). The 3D model includes, for example, coordinate information. The coordinate information may be, for example, two-dimensional coordinate information (x, y) consisting of width (x) and depth (y) values, or may be three-dimensional coordinate information (x, y, z) including a height (z) value. The configuration within the specific area includes, for example, not only the structure within the specific area but also the position and shape of objects located within the specific area. An example of generating the 3D model from the three-dimensional image information will be described below, but this is not limiting and known techniques can be used. First, the 3D model generation unit 12 performs modeling using a polygon mesh on the point cloud data. Then, the 3D model generation unit 12 places an image corresponding to the point cloud data on the surface of the polygon mesh.

[0030] Next, the virtual image generation unit 13 generates a virtual image of the specific area (S3). The virtual image is an image to be superimposed on a local image captured by a local terminal within the specific area. The local terminal will be described later. The virtual image generation unit 13 may generate, for example, a plurality of virtual images. Specifically, the virtual image includes, for example, a virtual image for each object in the specific area. The object is not particularly limited and can be set arbitrarily, and may be an artificial object or a natural object. Examples of the artificial object include a facility, an exhibition booth, a tenant, a classroom, etc. Examples of the natural object include a mountain, an ocean, a rock, etc. The virtual image for each object includes, for example, a position image of the object, and the position images are linked to each other for each object. The position image is, for example, an image related to the object. Generating the virtual image may mean, for example, reading the virtual image stored in memory 102 or storage device 104 in advance, or obtaining the virtual image from an external database via the communication network. Furthermore, the virtual information generating unit 13 may, for example, acquire information about the object (also referred to as object information) and generate the virtual image based on the object information.

[0031] Next, the presentation unit 14 presents the 3D model and the virtual image to the administrator (S4). The presentation format is not particularly limited, and may be, for example, by displaying the 3D model and the virtual image on the display device 106 of the device 10. Alternatively, the presentation unit 14 may present the 3D model and the virtual image by outputting them to the administrator's terminal via the communication network and displaying them on the display of the administrator's terminal. The administrator's terminal (also referred to as an administrator terminal) is not particularly limited, and may be, for example, a terminal of an administrator outside the specific area, such as a personal computer, a smartphone, a mobile phone, or a wearable terminal such as glasses or a wristwatch. Alternatively, the device 10 may function as the administrator terminal, for example. The presentation by the presentation unit 14 may be continuous, for example, while the local terminal displays a composite image, which will be described later.

[0032] Next, the coordinate information management unit 15 sets the coordinate information of the reference marker and the virtual image based on the position within the 3D model specified by the administrator, and manages the virtual image and the coordinate information of the virtual image in association with each other (S5). The administrator, for example, uses the input device 105 of the device 10 to specify any position within the 3D model as the position where the reference marker is to be placed and the position where the virtual image is to be displayed. The operation may also be performed using the administrator terminal instead of the input device 105 of the device 10. For example, the above-mentioned description can be used for the coordinate information. The position within the 3D model specified by the administrator may be, for example, a position where only the width and depth are specified, or a position where the height is also specified. The reference marker is, for example, a marker that triggers recording of the trajectory of the on-site terminal, as described below. Examples of the reference marker include a QR code (registered trademark), an AR marker, a barcode, and a Chameleon Code (registered trademark). The setting of each piece of coordinate information, particularly the coordinate information of the virtual image, by the coordinate information management unit 15 may be repeatedly performed, for example, while the local terminal is displaying a composite image, which will be described later.

[0033] Next, the image acquisition unit 11 acquires local captured images of the specific area captured by a local terminal (S6). The local terminal is not particularly limited and may be, for example, a terminal of a user in the specific area, such as a personal computer, a smartphone, a mobile phone, or a wearable terminal such as glasses or a wristwatch. The local captured images may be, for example, still images or videos. The acquisition is performed, for example, via the communication network. Specifically, the local captured images are, for example, two-dimensional images captured by a 2D camera (RGB camera) of the local terminal. The acquisition of the local captured images by the image acquisition unit 11 may be performed, for example, in parallel with and continuously with other processes.

[0034] Next, the local terminal information acquisition unit 16 acquires position information and orientation information of the local terminal relative to the reference marker (S7). Hereinafter, the position information and orientation information are collectively referred to as local terminal information. The position information and orientation information of the local terminal are, for example, information indicating the position and orientation of the local terminal recorded based on the reference markers arranged within the specific area and various sensors of the local terminal (such as an acceleration sensor or a geomagnetic sensor). The local terminal information acquisition unit 16 may, for example, acquire the position information and orientation information of the local terminal in association with the local captured image. The orientation information may also be, for example, the imaging direction of an imaging unit of the local terminal.

[0035] Then, the image information output unit 17 outputs a composite image in which the virtual image is superimposed on the captured image (S8), and the process ends (END). Specifically, first, based on the position information and orientation information of the local terminal, the image information output unit 17 refers to the 3D model in which the coordinate information of the fiducial marker and the virtual image is set, and determines whether a position corresponding to the coordinate information of the virtual image exists in the locally captured image. If the position exists, the image information output unit 17 outputs a composite image in which the virtual image is superimposed on the position to the local terminal. The output may be performed, for example, via the communication line network. If the specific area is indoors, the position of the local terminal cannot be identified using a global positioning system (GPS). However, according to this embodiment, by setting the fiducial marker, the position of the local terminal can be identified even indoors. The output of the composite image by the image information output unit 17 may be performed repeatedly, for example.

[0036] Furthermore, for example, in step (S4), after presenting the 3D model and the virtual image to the administrator, when the administrator specifies an arbitrary position within the 3D model, the presentation unit 14 may present to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to the coordinate information of the virtual image within the image included in the three-dimensional image information. The format of the presentation is not particularly limited and may be, for example, the same as described above. The "image included in the three-dimensional image information" may be an image having a position corresponding to the coordinate information of the virtual image, and is not particularly limited. For example, there may be multiple images. For example, the presentation unit 14 may generate multiple preview composite images using multiple images. For example, the administrator may refer to the preview composite image to re-specify and fine-tune the position where the reference marker is placed and the position where the virtual image is to be displayed. Then, after the position specification is confirmed, the processing of step (S5) is executed. This allows, for example, the administrator to specify the position while checking an example of the composite image displayed on the local terminal. Furthermore, by generating a plurality of composite preview images, it is possible to check how the virtual image appears from multiple directions.

[0037] The device 10 may further include, for example, a communication intermediation unit 18, as described above. For example, after step (S8), the communication intermediation unit 18 mediates communication between the administrator and the user having the local terminal (S9). Note that step (S9) is not limited to being performed after step (S8). Furthermore, step (S9) is an optional step and may not be executed. For example, the communication intermediation unit 18 may enable communication between the administrator terminal and the local terminal and mediate communication between the administrator and the user. The communication method is not particularly limited, and may include, for example, communication via text data such as chat, communication via voice data such as phone calls, etc. This allows communication between an administrator outside the specific area and a user within the specific area.

[0038] According to this embodiment, by using the 3D model, it is possible to more easily set the position of a virtual image to be superimposed on the on-site captured image. In other words, according to this embodiment, it is possible to set the position of a virtual image displayed using AR technology on a 3D model. This allows an administrator, for example, to associate the position where the virtual image is to be superimposed with any location within the specific area from outside the specific area where the on-site captured image is captured (e.g., a company, a home, etc.). Furthermore, according to this embodiment, for example, the administrator can mediate communication with a user within the specific area (e.g., "Please move to the location where the XX pin (virtual image) is displayed," "Please move along the arrow (virtual image)," etc.), allowing the administrator to navigate the user in real time.

[0039] [Embodiment 2] Fig. 4 is a block diagram showing an example of the configuration of an image processing system apparatus 100 of this embodiment. As shown in Fig. 4, this apparatus 100 includes the image processing apparatus 10 described in the first embodiment and a local terminal 20. Furthermore, this apparatus 100 may further include, for example, an administrator terminal 30. For example, there may be one local terminal 20 and one administrator terminal 30, or two or more local terminals 20 and 30. The image processing apparatus 10, the local terminal 20, and the administrator terminal 30 can communicate with each other via a communication network 40. The communication network 40 is the same as the communication network described above.

[0040] In the image processing device 10, the image acquisition unit 11 acquires the locally captured image captured by the local terminal 20, for example, in the same manner as described above. Also, the image information output unit 17 outputs the composite image to the local terminal 20, for example, in the same manner as described above.

[0041] The local terminal 20 is a terminal of a user who is in the specific area, and may be, for example, a personal computer (PC, for example, a desktop or notebook type), a smartphone, a tablet terminal, etc. The local terminal 20 is, for example, a device used in combination with the image processing device 10 described in the first embodiment.

[0042] The administrator terminal 30 is a terminal of an administrator who is outside the specific area, and may be, for example, a personal computer (PC, for example, desktop or notebook type), a smartphone, a tablet terminal, etc. The administrator terminal 30 is, for example, a device used in combination with the image processing device 10 described in the first embodiment.

[0043] There are no particular limitations on the configurations, including the hardware configurations, of the local terminal 20 and the administrator terminal 30. Furthermore, there are no particular limitations on the information processing methods in the local terminal 20 and the administrator terminal 30.

[0044] 5 is a block diagram showing an example of the configuration of the local terminal 20 of this embodiment. As shown in FIG. 5, the local terminal 20 includes, for example, an imaging unit 21 that captures the reference markers installed within the specific area and local captured images, a recording unit 22 that records, as position information, a trajectory of the reference marker after imaging, starting from the point where the reference marker is imaged, and records, as orientation information, the imaging direction in which the local captured images were captured, an output unit 23 that outputs the local captured images, the position information, and the orientation information to the image processing device 10, and a display processing unit 24 that acquires the composite image output from the image processing device 10 and displays it on a display. The local terminal 20 may also be capable of communicating with, for example, the administrator (administrator terminal 30).

[0045] The hardware configuration of the local terminal 20 is similar to the hardware configuration of the image processing device 10 shown in Figure 2, except that, for example, the central processing unit 101 functions as a recording unit 22, an output unit 23, and a display processing unit 24, and the imaging device 108 functions as an imaging unit 21.

[0046] Next, an example of an information processing method in the local terminal 20 will be described with reference to the flowchart of Fig. 6. The information processing method in the local terminal 20 (also referred to as a first information processing method) is implemented as follows, for example, using the local terminal 20 of Fig. 5. Note that the first information processing method is not limited to use of the local terminal 20 of Fig. 5.

[0047] First, the imaging unit 21 captures the fiducial markers installed within the specific area and on-site captured images (S21). Next, the recording unit 22 records the trajectory of the captured images, starting from the point where the fiducial markers were captured as position information, and records the capturing direction of the on-site captured images as orientation information (S22). Next, the output unit 23 outputs the on-site captured images, the position information, and the orientation information to the image processing device 10 (S23). Then, the display processing unit 24 acquires the composite image output from the image processing device 10 and displays it on the display (display device 106) (S24), and the process ends (END). Furthermore, the on-site terminal 20 may contact the administrator via the image processing device 10, for example, after the step (S24). Note that the contacting step is not limited to being performed after the step (S24). Furthermore, the contacting step is an optional step and may not be executed. Each step in the first information processing method may be executed repeatedly, for example.

[0048] 7 is a block diagram showing an example of the configuration of the administrator terminal 30 of this embodiment. As shown in FIG. 7, the administrator terminal 30 includes, for example, a display processing unit 31 that displays the 3D model and the virtual image presented from the image processing device 10, a designation unit 32 that designates the positions of the reference marker and the virtual image relative to the 3D model based on an operation by the administrator (for example, an operation using the input device 105), and an output unit 33 that outputs the designated positions to the image processing device 10. Furthermore, the display processing unit 31 may, for example, display the preview composite image presented from the image processing device 10. Furthermore, the administrator terminal 30 may, for example, be capable of contacting the user (local terminal 20).

[0049] The hardware configuration of the administrator terminal 30 is the same as the hardware configuration of the image processing device 10 shown in Figure 2, except that, for example, the central processing unit 101 functions as a display processing unit 31, a designation unit 32, and an output unit 33.

[0050] Next, an example of an information processing method in the administrator terminal 30 will be described based on the flowchart in Fig. 8. The information processing method in the administrator terminal 30 (also referred to as a second information processing method) is implemented, for example, as follows using the administrator terminal 30 in Fig. 7. Note that the second information processing method is not limited to use of the administrator terminal 30 in Fig. 7.

[0051] First, the display processing unit 31 displays the 3D model and the virtual image presented by the image processing device 10 (S31). Next, the designation unit 32 designates the positions of the reference marker and the virtual image relative to the 3D model based on an operation by the administrator (for example, an operation using the input device 105) (S32). Then, the output unit 33 outputs the designated positions to the image processing device 10 (S33), and the process ends (END). After the step (S32) and before the step (S33), the display processing unit 31 may display, for example, the preview composite image presented by the image processing device 10. Then, the designation unit 32 may designate the positions of the reference marker and the virtual image again after displaying the preview composite image. Furthermore, the administrator terminal 30 may contact the user via the image processing device 10 after the step (S33). Note that the contacting step is not limited to being performed after the step (S33). The contacting step is an optional step and may not be executed. Each step in the second information processing method may be executed repeatedly, for example.

[0052] The image processing method of this embodiment (also referred to as the second image processing method) includes the image processing method shown in FIG. 3, the first information processing method shown in FIG. 6, and the second information processing method shown in FIG.

[0053] According to this embodiment, similarly to the first embodiment, the position of the virtual image to be superimposed on the on-site captured image can be set more easily.

[0054] [Embodiment 3] An example of superimposing the virtual image on the captured image will be described.

[0055] 9(A) shows an example of a 3D model of a specific area in which coordinate information for each of the virtual images is set, and FIG. 9(B) shows an example of the positional relationship of each of the virtual images in the 3D model. As shown in FIG. 9, the image information output unit 17 specifies the imaging direction X of the local terminal 20, for example, based on the position information and orientation information of the local terminal 20. In the example of FIG. 9, the coordinate information for virtual images A to C exists in the imaging direction X.

[0056] FIG. 10 shows an example of a composite image output from the image processing device 10 to the local terminal 20, displayed on the display of the local terminal 20. As shown in FIG. 10, the image information output unit 17 generates a composite image by superimposing virtual images A to C (e.g., the position images; hereinafter, also referred to as pins) at positions in the local captured image corresponding to objects A to C present in the imaging direction X. The pins A to C may be superimposed, for example, based on the distance between the position information of the local terminal 20 and the coordinate information of the pins A to C, with pins farther from the local terminal 20 being smaller than the other pins. On the other hand, the pins A to C may be superimposed, for example, based on the distance between the position information of the local terminal 20 and the coordinate information of the pins A to C, with pins closer to the local terminal 20 being larger than the other pins. Furthermore, the outer edges of the pins may be colored, for example, according to the distance between the position information of the local terminal 20 and the coordinate information of the pins A to C.

[0057] 10, the detailed information image is specifically displayed in a window 1. In the figure, the window 1 is displayed below the on-site captured image, but this is merely an example and is not limiting. For example, when the present invention is applied to an exhibition, a detailed information image generated based on detailed information about the exhibited products and services, the company of the booth, etc., is displayed in the window 1.

[0058] [Embodiment 4] The program of this embodiment is a program for causing a computer to execute each step of the method of the present invention as a procedure. In the present invention, "procedure" may be read as "processing." The program of this embodiment may be recorded, for example, on a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples thereof include a read-only memory (ROM), a hard disk (HD), an optical disk, etc.

[0059] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0060] <Additional Notes> Some or all of the above embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) The system includes an image acquisition unit, a 3D model generation unit, a virtual image generation unit, a presentation unit, a coordinate information management unit, a local terminal information acquisition unit, and an image information output unit, the image acquisition unit acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera, the 3D model generation unit generates a 3D model showing a configuration within the specific area based on the three-dimensional image information; the virtual image generation unit generates a virtual image relating to the specific area; the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area, the presentation unit presents the 3D model and the virtual image to an administrator; the coordinate information management unit sets coordinate information of the reference marker and the virtual image based on a position in the 3D model specified by the administrator, and manages the virtual image and the coordinate information of the virtual image in association with each other; The image acquisition unit further acquires a local captured image captured within the specific area by a local terminal, the local terminal information acquisition unit acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker; The image information output unit refers to the 3D model based on the position information and orientation information of the local terminal, and determines whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, outputs a composite image with the virtual image superimposed at that position to the local terminal. (Appendix 2) The image processing device according to claim 1, wherein the presentation unit presents to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to coordinate information of the virtual image within an image included in the three-dimensional image information. (Appendix 3) It also includes a liaison department. 3. The image processing device according to claim 1, wherein the communication intermediary unit mediates communication between the administrator and the user having the local terminal. (Appendix 4) An image processing device according to any one of Supplementary Notes 1 to 3 and a local terminal, the image processing device and the on-site terminal are capable of communicating with each other via a communication network; In the image processing device, the image acquisition unit acquires a local captured image captured by the local terminal within the specific area; The image information output unit outputs the composite image to the local terminal. (Appendix 5) The method includes an image acquisition step, a 3D model generation step, a virtual image generation step, a presentation step, a coordinate information management step, a local terminal information acquisition step, and an image information output step. The image acquisition step acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera, the 3D model generation step generates a 3D model showing a configuration within the specific area based on the three-dimensional image information; the virtual image generating step generates a virtual image relating to the specific area; the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area, the presenting step presents the 3D model and the virtual image to an administrator; The coordinate information management step sets coordinate information of a reference marker and the virtual image based on a position in the 3D model specified by the administrator, and manages the virtual image and the coordinate information of the virtual image in association with each other; The image acquisition step further includes acquiring a local captured image of the specific area captured by a local terminal; the local terminal information acquisition step acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker; The image information output process is an image processing method in which, based on the position information and orientation information of the local terminal, the 3D model is referenced to determine whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, a composite image with the virtual image superimposed on that position is output to the local terminal. (Appendix 6) The image processing method according to claim 5, wherein the presentation step presents to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to the coordinate information of the virtual image within the image included in the three-dimensional image information. (Appendix 7) Further, it includes a liaison intermediation step, 7. The image processing method according to claim 5, wherein the communication intermediation step mediates communication between the administrator and the user having the local terminal. (Appendix 8) A program for causing a computer to execute procedures including an image acquisition procedure, a 3D model generation procedure, a virtual image generation procedure, a presentation procedure, a coordinate information management procedure, a local terminal information acquisition procedure, and an image information output procedure; The image acquisition step acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera, the 3D model generation step generates a 3D model showing a configuration within the specific area based on the three-dimensional image information; the virtual image generation step generates a virtual image relating to the specific area; the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area, the presentation step presents the 3D model and the virtual image to an administrator; the coordinate information management step includes setting coordinate information of a reference marker and the virtual image based on a position in the 3D model specified by the administrator, and managing the virtual image and the coordinate information of the virtual image in association with each other; The image acquisition step further includes acquiring a local captured image of the specific area captured by a local terminal; the local terminal information acquisition step acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker; The image information output procedure refers to the 3D model based on the position information and orientation information of the local terminal, and determines whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, outputs a composite image with the virtual image superimposed at that position to the local terminal. (Appendix 9) The program according to claim 8, wherein the presentation step presents to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to the coordinate information of the virtual image within the image included in the three-dimensional image information. (Appendix 10) Furthermore, it includes a contact procedure, 10. The program according to claim 8 or 9, wherein the communication intermediation procedure mediates communication between the administrator and the user having the local terminal. (Appendix 11) A computer-readable recording medium having recorded thereon a program according to any one of appendices 8 to 10. [Industrial Applicability]

[0061] According to the present invention, the position of the virtual image to be superimposed on the on-site captured image can be set more easily. Therefore, the present invention is particularly useful in fields that use AR technology, such as exhibitions and games. [Explanation of symbols]

[0062] 10 Image processing device 11 Image acquisition unit 12 3D model generation section 13 Virtual image generation unit 14 Presentation part 15 Coordinate information management department 16 Local terminal information acquisition unit 17 Image information output section 18 Liaison Department 20 Local terminal 21 Imaging unit 22 Recording section 23 Output section 24 Display processing section 30 Administrator terminal 31 Display processing section 32 Specified part 33 Output section 100 Image processing system device 101 Central Processing Unit 102 memory 103 Bus 104 Storage device 105 Input Device 106 Display device 107 Communication Devices 108 Imaging device 109 Sound Input / Output Device

Claims

1. The system includes an image acquisition unit, a 3D model generation unit, a virtual image generation unit, a presentation unit, a coordinate information management unit, a local terminal information acquisition unit, and an image information output unit, the image acquisition unit acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera; the 3D model generation unit generates a 3D model showing a configuration within the specific area based on the three-dimensional image information; the virtual image generation unit generates a virtual image relating to the specific area; the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area, the presentation unit presents the 3D model and the virtual image to an administrator; the coordinate information management unit sets coordinate information of the reference marker and the virtual image based on a position in the 3D model designated by the administrator as a position to place a reference marker and a position where the virtual image is to be displayed, and manages the virtual image and the coordinate information of the virtual image in association with each other; The image acquisition unit further acquires a local captured image captured within the specific area by a local terminal, the local terminal information acquisition unit acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker; The image information output unit refers to the 3D model based on the position information and orientation information of the local terminal, and determines whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, outputs a composite image with the virtual image superimposed on the position to the local terminal.

2. The image processing device according to claim 1 , wherein the presentation unit presents to the administrator a preview composite image in which the virtual image is superimposed at a position corresponding to coordinate information of the virtual image within the image included in the three-dimensional image information.

3. It also includes a liaison department. The image processing apparatus according to claim 1 , wherein the communication intermediary unit mediates communication between the administrator and the users who have the local terminals.

4. An image processing device according to any one of claims 1 to 3 and a local terminal, the image processing device and the on-site terminal are capable of communicating with each other via a communication network; In the image processing device, the image acquisition unit acquires a local captured image captured by the local terminal within the specific area; The image information output unit outputs the composite image to the local terminal.

5. The method includes an image acquisition step, a 3D model generation step, a virtual image generation step, a presentation step, a coordinate information management step, a local terminal information acquisition step, and an image information output step, The image acquisition step acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera, the 3D model generation step generates a 3D model showing a configuration within the specific area based on the three-dimensional image information; the virtual image generating step generates a virtual image relating to the specific area; the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area, the presenting step presents the 3D model and the virtual image to an administrator; the coordinate information management step sets coordinate information of the reference marker and the virtual image based on a position in the 3D model designated by the administrator as a position to place a reference marker and a position where the virtual image is to be displayed, and manages the virtual image and the coordinate information of the virtual image in association with each other; The image acquisition step further includes acquiring a local captured image of the specific area captured by a local terminal; the local terminal information acquisition step acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker; The image information output process refers to the 3D model based on the position information and orientation information of the local terminal, determines whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, outputs a composite image with the virtual image superimposed on the position to the local terminal.This is an image processing method in which each process is executed by a computer.

6. 6. The image processing method according to claim 5, wherein the presenting step presents to the manager a preview composite image in which the virtual image is superimposed at a position corresponding to the coordinate information of the virtual image within the image included in the three-dimensional image information.

7. Further, it includes a liaison intermediation step, 7. The image processing method according to claim 5, wherein said communication intermediation step mediates communication between said administrator and said users having said local terminals.

8. a program for causing a computer to execute procedures including an image acquisition procedure, a 3D model generation procedure, a virtual image generation procedure, a presentation procedure, a coordinate information management procedure, a local terminal information acquisition procedure, and an image information output procedure; The image acquisition step acquires three-dimensional image information obtained by capturing an image of a specific area with a 3D camera, the 3D model generation step generates a 3D model showing a configuration within the specific area based on the three-dimensional image information; the virtual image generation step generates a virtual image relating to the specific area; the virtual image is an image to be superimposed on a local captured image captured by a local terminal within the specific area, the presentation step presents the 3D model and the virtual image to an administrator; the coordinate information management step includes setting coordinate information of the reference marker and the virtual image based on a position in the 3D model designated by the administrator as a position to place a reference marker and a position where the virtual image is to be displayed, and managing the virtual image and the coordinate information of the virtual image in association with each other; The image acquisition step further includes acquiring a local captured image of the specific area captured by a local terminal; the local terminal information acquisition step acquires position information of the local terminal and orientation information of the local terminal relative to the reference marker; The image information output procedure refers to the 3D model based on the position information and orientation information of the local terminal, and determines whether a position corresponding to the coordinate information of the virtual image exists in the local captured image, and if so, outputs a composite image with the virtual image superimposed at that position to the local terminal.

9. 9. The program according to claim 8, wherein the step of presenting comprises presenting to the administrator a composite preview image in which the virtual image is superimposed at a position corresponding to coordinate information of the virtual image within an image included in the three-dimensional image information.

10. Furthermore, it includes a contact procedure, 10. The program according to claim 8, wherein the communication intermediation procedure mediates communication between the administrator and the user having the local terminal.

Images