Image processing device, image processing system device, image processing method, program, and recording medium

Abstract

To provide an image processing apparatus configured to allow a user to recognize a virtual image corresponding to an object outside a captured image, an image processing system apparatus, an image processing method, a program, and a recording medium.SOLUTION: An image processing apparatus 10 includes: an imaging information acquisition unit 11 which acquires a captured image of a specific area captured by a user terminal; a user terminal information acquisition unit 14 which acquires position information and direction information of the user terminal; a specifying unit 15 which specifies an object existing in the captured image, and specifies a direction of an object that does not exist in the captured image, based on the user terminal; and an image information output unit 16 which superimposes a position image for the object that exists in the captured image in a position corresponding to the object in the captured image, sets an outside object display section in the captured image, for the object that does not exist in the captured image, and outputs to the user terminal, an image obtained by superimposing a part or all of the position image of the object on the outside object display section, to the user terminal, in accordance with the direction of the object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an image processing system, an image processing method, a program, and a recording medium. [Background technology]

[0002] BACKGROUND ART Techniques have been reported in which a virtual image is superimposed on a captured image using AR (Augmented Reality) technology (for example, Patent Document 1, etc.). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-006582 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technique of Patent Document 1 has a problem in that the user cannot recognize a virtual image corresponding to an object outside the captured image.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device, an image processing system, an image processing method, a program, and a recording medium that allow a user to recognize a virtual image corresponding to an object outside a captured image. [Means for solving the problem]

[0006] In order to achieve the above object, the image processing device of the present invention comprises: The system includes an imaging information acquisition unit, a virtual information generation unit, a map information management unit, a user terminal information acquisition unit, an identification unit, and an image information output unit, the imaging information acquisition unit acquires captured images captured by user terminals in a specific area; the virtual information generation unit generates a virtual image relating to the specific area; the map information management unit manages position information of the object by linking it to the map of the specific area; the user terminal information acquisition unit acquires location information and orientation information of the user terminal, the identification unit identifies an object that is present in the captured image and an object that is not present in the captured image based on the map, position information of the user terminal, and orientation information of the user terminal, and, for an object that is not present in the captured image, identifies a direction in which the object is located relative to the user terminal based on the map, position information of the user terminal, and orientation information of the user terminal; the image information output unit outputs a composite image in which the virtual image is superimposed on the captured image; the virtual image generated by the virtual information generation unit includes a virtual image for each of the objects in the specific area, the virtual image for each of the objects includes a position image of the object, and the position images are linked to each other for each of the objects; the image information output unit superimposes the position image at a position corresponding to the object present in the captured image; and For objects that do not exist within the captured image, an out-of-range object display unit is provided within the captured image, and an image in which part or all of the position image of the object is superimposed on the out-of-range object display unit is output to the user terminal depending on the direction in which the object is located.

[0007] The image processing method of the present invention comprises: The method includes an imaging information acquisition step, a virtual information generation step, a map information management step, a user terminal information acquisition step, a specification step, and an image information output step, The imaging information acquisition step acquires an image captured by a user terminal in a specific area, the virtual information generating step generates a virtual image relating to the specific area; the map information management step links and manages position information of the object on a map of the specific area; the user terminal information acquisition step acquires position information and orientation information of the user terminal; the identification step includes identifying an object that is present in the captured image and an object that is not present in the captured image based on the map, position information of the user terminal, and orientation information of the user terminal, and, for an object that is not present in the captured image, identifying a direction in which the object is located relative to the user terminal based on the map, position information of the user terminal, and orientation information of the user terminal; the image information output step outputs a composite image in which the virtual image is superimposed on the captured image, the virtual image generated in the virtual information generating step includes a virtual image for each of the objects in the specific area, the virtual image for each of the objects includes a position image of the object, and the position images are linked to each other for each of the objects; The image information output step includes superimposing the position image on the object present in the captured image at a position corresponding to the object in the captured image; and For objects that do not exist in the captured image, an out-of-range object display unit is provided within the captured image, and an image in which part or all of the position image of the object is superimposed on the out-of-range object display unit is output to the user terminal depending on the direction in which the object is located. [Effects of the Invention]

[0008] According to the present invention, in addition to a virtual image corresponding to an object within a captured image, a user can recognize a virtual image corresponding to an object outside the captured image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image processing apparatus according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of the image processing apparatus according to the first embodiment. [Figure 3] FIG. 3 is a flowchart showing an example of processing in the image processing apparatus of the first embodiment. [Figure 4]FIG. 4 is a schematic diagram showing an example of a map of a specific area to which the position information of objects is linked. [Figure 5] FIG. 5 is a schematic diagram showing an example of an image output from the image processing apparatus according to the second embodiment to a user terminal, displayed on the display of the user terminal. [Figure 6] FIG. 6 is a block diagram showing an example of the configuration of an image processing system apparatus according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] In the image processing device of the present invention, for example, The image information output section may be configured to provide a predetermined range including a part or the whole of at least one end of the captured image as the out-of-range object display section.

[0011] In the image processing device of the present invention, for example, the map information management unit calculates a distance between the user terminal and the object on the map based on position information of the user terminal and position information of the object; The image information output unit may be configured to superimpose a part or all of the position images of the objects on the out-of-range object display unit so that the objects are arranged in one direction in ascending order of distance.

[0012] In the image processing device of the present invention, for example, the virtual image for each object further includes a detailed information image of the object, and the position image and the detailed information image are linked to each other for each object; The image information output unit may be configured to provide a window portion within the captured image and superimpose the detailed information image linked to the position image of the object present in the captured image onto the window portion.

[0013] The image processing system of the present invention comprises: The apparatus includes the image processing device of the present invention and a user terminal. The image processing device and the user terminal can communicate with each other via a communication network.

[0014] In the image processing method of the present invention, for example, The image information output step may be implemented in a manner that a predetermined range including a part or the whole of at least one end of the captured image is provided as the out-of-range object display section.

[0015] In the image processing method of the present invention, for example, the map information management step calculates a distance between the user terminal and the object on the map based on position information of the user terminal and position information of the object; The image information output step may be performed in a manner in which part or all of the position images of the objects are superimposed on the out-of-range object display unit so that the objects are arranged in one direction in ascending order of the distance.

[0016] In the image processing method of the present invention, for example, the virtual image for each object further includes a detailed information image of the object, and the position image and the detailed information image are linked to each other for each object; The image information output step may be performed in a manner that a window portion is provided within the captured image, and the detailed information image linked to a position image of the object present in the captured image is superimposed on the window portion.

[0017] The program of the present invention is a program for causing a computer to execute the steps of the method of the present invention as a procedure.

[0018] The recording medium of the present invention is a computer-readable recording medium on which the program of the present invention is recorded.

[0019] Next, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiments. In the following drawings, the same parts are denoted by the same reference numerals. Furthermore, the descriptions of the embodiments can be mutually incorporated unless otherwise specified, and the configurations of the embodiments can be combined unless otherwise specified.

[0020] [Embodiment 1] 1 is a block diagram showing an example of the configuration of an image processing device 10 according to this embodiment. As shown in FIG. 1, the device 10 includes an imaging information acquisition unit 11, a virtual information generation unit 12, a map information management unit 13, a user terminal information acquisition unit 14, an identification unit 15, and an image information output unit 16.

[0021] The device 10 may be, for example, a single device including the above-described units, or a device in which the units can be connected via a communication network. The device 10 can also be connected to an external device (described later) via the communication network. The communication network is not particularly limited and any known network can be used, for example, a wired or wireless network. Examples of the communication network include the Internet, the World Wide Web (WWW), a telephone line, a Local Area Network (LAN), a Storage Area Network (SAN), a Delay Tolerant Networking (DTN), a Low Power Wide Area Network (LPWA), and a Local 5G (L5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), Local 5G, and LPWA. Examples of the wireless communication include direct communication between devices (Ad Hoc communication), infrastructure communication, and indirect communication via an access point. The device 10 may be incorporated into a server as a system. The device 10 may be, for example, a personal computer (PC, for example, a desktop or notebook type) on which the program of the present invention is installed, a smartphone, a tablet terminal, etc. Furthermore, the device 10 may be in the form of cloud computing or edge computing, for example, in which at least one of the units is located on a server and the other units are located on a terminal.

[0022] 2 shows an example of a block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a central processing unit (CPU, GPU, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, a display device 106, a communication device 107, an imaging device 108, and an audio input / output device 109. The components of the device 10 are connected to each other via the bus 103 and their respective interfaces (I / F).

[0023] The central processing unit 101 is responsible for overall control of the device 10. In the device 10, the central processing unit 101 executes, for example, the program of the present invention and other programs, and also reads and writes various types of information. Specifically, for example, the central processing unit 101 functions as an imaging information acquisition unit 11, a virtual information generation unit 12, a map information management unit 13, a user terminal information acquisition unit 14, an identification unit 15, and an image information output unit 16.

[0024] The bus 103 can also be connected to, for example, an external device. Examples of the external device include an external storage device (such as an external database), a printer, an external input device, an external display device, and an external imaging device. The device 10 can be connected to an external network (the communication line network) by, for example, a communication device 107 connected to the bus 103, and can also be connected to other devices via the external network.

[0025] The memory 102 may be, for example, a main memory (primary storage device). When the central processing unit 101 performs processing, the memory 102 reads various operating programs, such as the program of the present invention, stored in the storage device 104 (described later), and the central processing unit 101 receives data from the memory 102 and executes the programs. The main memory may be, for example, a RAM (random access memory). Alternatively, the memory 102 may be, for example, a ROM (read only memory).

[0026] The storage device 104 is also referred to as an auxiliary storage device, for example, in contrast to the main memory (primary storage device). As described above, the storage device 104 stores an operating program including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive that reads and writes data from and to the recording medium. The recording medium is not particularly limited and may be, for example, an internal or external type, such as a hard disk (HD), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, or memory card. The storage device 104 may be, for example, a hard disk drive (HDD) or a solid state drive (SSD) in which the recording medium and drive are integrated.

[0027] In the present device 10, the memory 102 and the storage device 104 can also store various information such as log information, information acquired from an external database (not shown) or an external device, information generated by the present device 10, and information used when the present device 10 executes processing. Note that at least a portion of the information may be stored, for example, in an external server other than the memory 102 and the storage device 104, or may be stored in a distributed manner across multiple terminals using blockchain technology or the like.

[0028] The device 10 may further include, for example, an input device 105, a display device 106, an imaging device 108, and an audio input / output device 109. The input device 105 is, for example, a touch panel, a keyboard, a mouse, etc. The display device 106 is, for example, an LED display, a liquid crystal display, etc. The imaging device 108 is, for example, a 2D camera, a 3D camera, an infrared camera, etc. The audio input / output device 109 is, for example, a microphone, a speaker, etc.

[0029] Next, an example of the image processing method of this embodiment will be described with reference to the flowchart of Fig. 3. The image processing method of this embodiment is implemented as follows, for example, using the image processing device 10 of Fig. 1. Note that the image processing method of this embodiment is not limited to use of the image processing device 10 of Fig. 1.

[0030] First, the imaging information acquisition unit 11 acquires captured images captured by a user terminal in a specific area (S1). The specific area is not particularly limited and may be indoors or outdoors. Examples of indoor locations include exhibition halls, event venues, department stores, and schools such as universities. The user terminal is a user's terminal and is not particularly limited and may be, for example, a personal computer, a smartphone, a mobile phone, or a wearable device such as glasses or a wristwatch. The captured images may be, for example, still images or videos. The acquisition is performed, for example, via the communication network.

[0031] Next, the virtual information generation unit 12 generates a virtual image of the specific area (S2). The generated virtual image includes a virtual image of each object in the specific area. The objects are not particularly limited and can be set arbitrarily, and may be man-made or natural. Examples of man-made objects include facilities, exhibition booths, tenants, classrooms, etc. Examples of natural objects include mountains, oceans, rocks, etc. The virtual image for each object includes a position image of the object, and the position images are linked to each other for each object. The position image is, for example, an image related to the object. Generating the virtual image may mean, for example, reading the virtual image stored in memory 102 or storage device 104 in advance, or may mean acquiring the virtual image from an external database via the communication network. Furthermore, the virtual information generation unit 12 may, for example, acquire information about the object (also referred to as object information) and generate the virtual image based on the object information.

[0032] The identification unit 15, which will be described later, may identify a specific area from the captured image, for example, before the step (S2). The method for identifying the specific area is not particularly limited, and for example, a known technique may be used. Then, in the step (S2), the virtual information generation unit 12 may generate a virtual image relating to the specific area identified by the identification unit 15.

[0033] Next, the map information management unit 13 links and manages the position information of the object on a map of the specific area (S3). The position information of the object is, for example, information indicating the position of the object. The map may be a map that represents the specific area in two dimensions or a map that represents the specific area in three dimensions.

[0034] Next, the user terminal information acquisition unit 14 acquires location information and orientation information of the user terminal (S4). The location information and orientation information of the user terminal are, for example, information indicating the location and orientation of the user terminal identified by a location code or orientation mark placed within the specific area, various sensors of the user terminal, etc. The location information of the user terminal and the orientation information of the user terminal are collectively referred to as user terminal information. The location code may also serve as the orientation mark, for example. The location code and the orientation mark may be, for example, a QR code (registered trademark), an AR marker, a barcode, a chameleon code (registered trademark), etc. The user terminal information acquisition unit 14 may acquire the location information and orientation information of the user terminal by linking them to the captured image, for example. The orientation information may also be, for example, the imaging direction of an imaging device of the user terminal.

[0035] Next, the identification unit 15 identifies objects that are present in the captured image and objects that are not present based on the map, the position information of the user terminal, and the orientation information of the user terminal (S5a). Furthermore, for objects that are not present in the captured image, the identification unit 15 identifies the direction in which the object is located relative to the user terminal based on the map, the position information of the user terminal, and the orientation information of the user terminal (S5b). The direction is, for example, at least one of up, down, left, and right relative to the user terminal.

[0036] The image information output unit 16 then outputs a composite image in which the virtual image is superimposed on the captured image (S6), and the process ends (END). The output may be performed, for example, via the communication network. Specifically, the image information output unit 16 superimposes the position image of the object present in the captured image at a position corresponding to the object in the captured image. Furthermore, the image information output unit 16 provides an out-of-range object display unit in the captured image for the object not present in the captured image, and outputs an image to the user terminal in which a portion or all of the position image of the object is superimposed on the out-of-range object display unit depending on the direction in which the object is located. In other words, the image information output unit 16 can be said to superimpose the position image of the object on the out-of-range object display unit, for example, reflecting the positional relationship between the user terminal and the object. The portion of the position image of the object is not particularly limited, and may be, for example, 1 / 2, 2 / 3, etc., when the entire position image is 1. The image information output unit 16 may provide, for example, a predetermined range including a part or all of at least one edge of the captured image as the out-of-range object display unit. The predetermined range is not particularly limited and can be set arbitrarily. For example, when the range from the edge of the captured image to the center of the captured image is 1, the predetermined range may be 1 / 3, 1 / 4, 1 / 5, etc.

[0037] Furthermore, for example, before the step (S6), the map information management unit 13 may calculate the distance between the user terminal and the object on the map based on the position information of the user terminal and the position information of the object. Then, for example, in the step (S6), the image information output unit 16 may superimpose a part or all of the position image of the object on the out-of-range object display unit so that the objects are arranged in one direction in ascending order of the distance. The one direction is not particularly limited and may be, for example, from top to bottom, bottom to top, right to left, left to right, etc.

[0038] Furthermore, the virtual image for each object generated by the virtual information generating unit 12 may further include a detailed information image of the object. In this case, the position image and the detailed information image of the object are linked to each other for each object. Then, for example, in the step (S6), the image information output unit 16 further provides a window section within the captured image and superimposes the detailed information image linked to the position image of the object present in the captured image on the window section. The window section can display, for example, multiple detailed information images on a screen that allows transitions between them. The detailed information image is, for example, an image with a hierarchical structure of one or more layers. When the detailed information image is an image with a hierarchical structure of one or more layers, the window section can display, for example, the detailed information image on a screen that allows transitions between images for each layer. Specifically, the detailed information image is, for example, an image based on detailed information of the object, and one detailed information image or two detailed information images may be linked to one position image of the object.

[0039] According to this embodiment, by providing the out-of-range object display unit in the captured image, in addition to the virtual image corresponding to the object in the captured image, a virtual image corresponding to the object outside the captured image can be superimposed on the captured image. Therefore, the user can, for example, recognize the virtual image corresponding to the object outside the captured image with a seamless operation. Furthermore, by setting a predetermined range including the edge of the captured image as the out-of-range object display unit, it is possible to superimpose the virtual image corresponding to the object outside the captured image on the captured image without affecting the visibility of the position image of the object present in the captured image.

[0040] [Embodiment 2] An example of superimposing a position image of the object on the captured image will be described with reference to FIGS.

[0041] Fig. 4 shows an example of a map of a specific area to which the position information of the objects is linked. Note that in Fig. 4, the map is shown as a two-dimensional map, but is not limited to this and may be a three-dimensional map, for example. As shown in Fig. 4, the identification unit 15, for example, identifies an imaging direction X of the user terminal 1, identifies objects A to C as objects present in the imaging direction X, and identifies objects D to O as objects not present in the imaging direction X. Furthermore, the identification unit 15 identifies in which direction the objects D to O are located when the imaging direction X of the user terminal 1 is used as a reference.

[0042] Fig. 5 shows an example in which an image output from the image processing device 10 to the user terminal 1 is displayed on the display of the user terminal 1. Fig. 5(A) is an example in which a predetermined range including a part of the edge of the captured image is set as the out-of-range object display area (the area indicated by diagonal lines), and Fig. 5(B) is an example in which a predetermined range including the entire edge of the captured image is set as the out-of-range object display area (the area indicated by diagonal lines).

[0043] As shown in Fig. 5, position images of objects A to C present in the imaging direction X (hereinafter, the position images of the objects are also referred to as pins) are displayed superimposed on the corresponding positions of the captured image. The pins A to C of the objects A to C may be superimposed, for example, based on the distance between the user terminal 1 and the objects A to C, with pins farther from the user terminal 1 being smaller than the other pins. On the other hand, the pins A to C of the objects A to C may be superimposed, for example, based on the distance between the user terminal 1 and the objects A to C, with pins closer to the user terminal 1 being larger than the other pins. Furthermore, the outer edges of the pins may be colored according to the distance between the user terminal 1 and the objects A to C, for example.

[0044] 5, position images of objects D to O that are not present in the imaging direction X are displayed superimposed on the out-of-range object display section 3. In FIG. 5(B), pins of the objects are superimposed on the out-of-range object display section 3 according to the positional relationship between the user terminal 1 and the objects. On the other hand, in FIG. 5(A), in addition to the positional relationship in FIG. 5(B), pins of the objects are superimposed on the out-of-range object display section 3 according to the distance between the user terminal 1 and the objects.

[0045] 5, the detailed information image is specifically displayed in a window 4. In the figure, the window 4 is displayed below the captured image, but this is merely an example and is not limiting. For example, when the present invention is applied to an exhibition, a detailed information image generated based on detailed information about the exhibited products and services, the company operating the booth, etc., is displayed in the window 4.

[0046] [Embodiment 3] Fig. 6 is a block diagram showing an example of the configuration of an image processing system device 100 of this embodiment. As shown in Fig. 6, this device 100 includes the image processing device 10 described in the first embodiment and a user terminal 20. The image processing device 10 and the user terminal 20 can communicate with each other via a communication network 30. The communication network 30 is the same as the communication network described above. The number of user terminals 20 is not particularly limited and may be one or more.

[0047] The user terminal 20 is not particularly limited, and may be, for example, a personal computer (PC, for example, desktop or notebook type), a smartphone, a tablet terminal, or the like.

[0048] [Embodiment 4] The program of this embodiment is a program for causing a computer to execute each step of the method of the present invention as a procedure. In the present invention, "procedure" may be read as "processing." The program of this embodiment may be recorded, for example, on a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples thereof include a read-only memory (ROM), a hard disk (HD), an optical disk, etc.

[0049] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0050] <Additional Notes> Some or all of the above embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) The system includes an imaging information acquisition unit, a virtual information generation unit, a map information management unit, a user terminal information acquisition unit, an identification unit, and an image information output unit, the imaging information acquisition unit acquires captured images captured by user terminals in a specific area; the virtual information generation unit generates a virtual image relating to the specific area; the map information management unit manages position information of the object by linking it to the map of the specific area; the user terminal information acquisition unit acquires location information and orientation information of the user terminal, the identification unit identifies an object that is present in the captured image and an object that is not present in the captured image based on the map, position information of the user terminal, and orientation information of the user terminal, and, for an object that is not present in the captured image, identifies a direction in which the object is located relative to the user terminal based on the map, position information of the user terminal, and orientation information of the user terminal; the image information output unit outputs a composite image in which the virtual image is superimposed on the captured image; the virtual image generated by the virtual information generation unit includes a virtual image for each of the objects in the specific area, the virtual image for each of the objects includes a position image of the object, and the position images are linked to each other for each of the objects; the image information output unit superimposes the position image at a position corresponding to the object present in the captured image; and An image processing device that provides an out-of-range object display unit within the captured image for objects that do not exist within the captured image, and outputs to the user terminal an image in which part or all of the position image of the object is superimposed on the out-of-range object display unit depending on the direction in which the object is located. (Appendix 2) 2. The image processing device according to claim 1, wherein the image information output unit provides a predetermined range including a part or all of at least one end of the captured image as the out-of-range object display unit. (Appendix 3) the map information management unit calculates a distance between the user terminal and the object on the map based on position information of the user terminal and position information of the object; 3. The image processing device according to claim 1, wherein the image information output unit superimposes part or all of the position images of the objects on the out-of-range object display unit so that the objects are arranged in one direction in order of shortest distance. (Appendix 4) the virtual image for each object further includes a detailed information image of the object, and the position image and the detailed information image are linked to each other for each object; The image processing device according to any one of claims 1 to 3, wherein the image information output unit provides a window portion within the captured image and superimposes the detailed information image linked to the position image of the object present in the captured image on the window portion. (Appendix 5) An image processing device according to any one of Supplementary Notes 1 to 4, and a user terminal; The image processing system apparatus is configured such that the image processing apparatus and the user terminal can communicate with each other via a communication network. (Appendix 6) The method includes an imaging information acquisition step, a virtual information generation step, a map information management step, a user terminal information acquisition step, a specification step, and an image information output step, The imaging information acquisition step acquires an image captured by a user terminal in a specific area, the virtual information generating step generates a virtual image relating to the specific area; the map information management step links and manages position information of the object on a map of the specific area; the user terminal information acquisition step acquires position information and orientation information of the user terminal; the identification step includes identifying an object that is present in the captured image and an object that is not present in the captured image based on the map, position information of the user terminal, and orientation information of the user terminal, and, for an object that is not present in the captured image, identifying a direction in which the object is located relative to the user terminal based on the map, position information of the user terminal, and orientation information of the user terminal; the image information output step outputs a composite image in which the virtual image is superimposed on the captured image, the virtual image generated in the virtual information generating step includes a virtual image for each of the objects in the specific area, the virtual image for each of the objects includes a position image of the object, and the position images are linked to each other for each of the objects; The image information output step includes superimposing the position image on the object present in the captured image at a position corresponding to the object in the captured image; and An image processing method in which, for an object that does not exist in the captured image, an out-of-range object display unit is provided in the captured image, and an image in which part or all of the position image of the object is superimposed on the out-of-range object display unit depending on the direction in which the object is located is output to the user terminal. (Appendix 7) 7. The image processing method according to claim 6, wherein the image information output step provides a predetermined range including a part or all of at least one end of the captured image as the out-of-range object display section. (Appendix 8) the map information management step calculates a distance between the user terminal and the object on the map based on position information of the user terminal and position information of the object; The image processing method according to claim 6 or 7, wherein the image information output step superimposes part or all of the position images of the objects in the out-of-range object display unit so that the objects are arranged in one direction in order of shortest distance. (Appendix 9) the virtual image for each object further includes a detailed information image of the object, and the position image and the detailed information image are linked to each other for each object; An image processing method according to any one of Appendices 6 to 8, wherein the image information output step provides a window portion within the captured image and superimposes the detailed information image linked to a position image of the object present in the captured image on the window portion. (Appendix 10) a program for causing a computer to execute procedures including an imaging information acquisition procedure, a virtual information generation procedure, a map information management procedure, a user terminal information acquisition procedure, a specification procedure, and an image information output procedure; The imaging information acquisition step acquires an image captured by a user terminal in a specific area, The virtual information generation step generates a virtual image relating to the specific area; the map information management step includes managing position information of an object by linking it to a map of the specific area; the user terminal information acquisition step acquires location information and orientation information of the user terminal; the identification step includes identifying an object that is present in the captured image and an object that is not present in the captured image based on the map, position information of the user terminal, and orientation information of the user terminal, and, for an object that is not present in the captured image, identifying a direction in which the object is located relative to the user terminal based on the map, position information of the user terminal, and orientation information of the user terminal; the image information output step includes outputting a composite image in which the virtual image is superimposed on the captured image; the virtual image generated in the virtual information generation step includes a virtual image for each of the objects in the specific area, the virtual image for each of the objects includes a position image of the object, and the position images are linked to each other for each of the objects; The image information output step includes superimposing the position image on the object present in the captured image at a position corresponding to the object in the captured image; and For objects that do not exist within the captured image, an out-of-range object display unit is provided within the captured image, and an image in which part or all of the object's position image is superimposed on the out-of-range object display unit is output to the user terminal depending on the direction in which the object is located. (Appendix 11) The program according to claim 10, wherein the image information output procedure provides a predetermined range including a part or all of at least one end of the captured image as the out-of-range object display section. (Appendix 12) the map information management step calculates a distance between the user terminal and the object on the map based on position information of the user terminal and position information of the object; 12. The program according to claim 10, wherein the image information output step superimposes part or all of the position image of the object in the out-of-range object display unit so that the objects are arranged in one direction in order of shortest distance. (Appendix 13) the virtual image for each object further includes a detailed information image of the object, and the position image and the detailed information image are linked to each other for each object; The program described in any one of Appendices 10 to 12, wherein the image information output procedure provides a window section within the captured image and superimposes the detailed information image linked to the position image of the object present in the captured image on the window section. (Appendix 14) A computer-readable recording medium having recorded thereon a program according to any one of appendices 10 to 13. [Industrial Applicability]

[0051] According to the present invention, a user can recognize a virtual image corresponding to an object outside a captured image. Therefore, the present invention is particularly useful in fields that use AR technology, such as exhibitions and games. [Explanation of symbols]

[0052] 1. User terminal 3 Out-of-range object display section 4 Window section 10 Image processing device 11. Imaging information acquisition unit 12 Virtual Information Generation Unit 13 Map Information Management Department 14 User terminal information acquisition unit 15 Specific section 16 Image information output section 20 User terminal 100 Image processing system device 101 Central Processing Unit 102 memory 103 Bus 104 Storage device 105 Input Device 106 Display device 107 Communication Devices 108 Imaging device 109 Sound Input / Output Device

Claims

1. The system includes an imaging information acquisition unit, a virtual information generation unit, a map information management unit, a user terminal information acquisition unit, an identification unit, and an image information output unit, the imaging information acquisition unit acquires captured images captured by user terminals in a specific area; the virtual information generation unit generates a virtual image relating to the specific area; the map information management unit manages position information of the object by linking it to the map of the specific area; the user terminal information acquisition unit acquires location information and orientation information of the user terminal, the identification unit identifies an object that is present in the captured image and an object that is not present in the captured image based on the map, position information of the user terminal, and orientation information of the user terminal, and, for an object that is not present in the captured image, identifies a direction in which the object is located relative to the user terminal based on the map, position information of the user terminal, and orientation information of the user terminal; the image information output unit outputs a composite image in which the virtual image is superimposed on the captured image; the virtual image generated by the virtual information generation unit includes a virtual image for each of the objects in the specific area, the virtual image for each of the objects includes a position image of the object, and the position images are linked to each other for each of the objects; the image information output unit superimposes the position image on a position corresponding to the object present in the captured image; and providing an out-of-range object display unit in the captured image for the object that does not exist in the captured image, and outputting an image in which a part or all of a position image of the object is superimposed on the out-of-range object display unit according to a direction in which the object is located to the user terminal; the map information management unit calculates a distance between the user terminal and the object on the map based on position information of the user terminal and position information of the object; the image information output unit superimposes a part or all of the position images of the objects on the out-of-range object display unit so that the objects are arranged in one direction in ascending order of the distance; Image processing device.

2. The image processing device according to claim 1 , wherein the image information output section provides a predetermined range including a part or all of at least one edge of the captured image as the out-of-range object display section.

3. the virtual image for each object further includes a detailed information image of the object, and the position image and the detailed information image are linked to each other for each object; 3. The image processing device according to claim 1, wherein the image information output unit provides a window portion within the captured image and superimposes the detailed information image linked to the position image of the object present in the captured image on the window portion.

4. An image processing device according to any one of claims 1 to 3 and a user terminal, The image processing system apparatus is configured such that the image processing apparatus and the user terminal can communicate with each other via a communication network.

5. The method includes an imaging information acquisition step, a virtual information generation step, a map information management step, a user terminal information acquisition step, a specification step, and an image information output step, The imaging information acquisition step acquires an image captured by a user terminal in a specific area, the virtual information generating step generates a virtual image relating to the specific area; the map information management step links and manages position information of the object on a map of the specific area; the user terminal information acquisition step acquires position information and orientation information of the user terminal; the identification step includes identifying an object that is present in the captured image and an object that is not present in the captured image based on the map, position information of the user terminal, and orientation information of the user terminal, and, for an object that is not present in the captured image, identifying a direction in which the object is located relative to the user terminal based on the map, position information of the user terminal, and orientation information of the user terminal; the image information output step outputs a composite image in which the virtual image is superimposed on the captured image, the virtual image generated in the virtual information generating step includes a virtual image for each of the objects in the specific area, the virtual image for each of the objects includes a position image of the object, and the position images are linked to each other for each of the objects; The image information output step includes superimposing the position image on the object present in the captured image at a position corresponding to the object in the captured image; and providing an out-of-range object display unit in the captured image for the object that does not exist in the captured image, and outputting an image in which a part or all of a position image of the object is superimposed on the out-of-range object display unit according to a direction in which the object is located to the user terminal; the map information management step calculates a distance between the user terminal and the object on the map based on position information of the user terminal and position information of the object; the image information output step includes superimposing, on the out-of-range object display unit, a part or all of the position images of the objects so as to be arranged in one direction in ascending order of the distance; An image processing method in which each step is performed by a computer.

6. 6. The image processing method according to claim 5, wherein the image information output step provides a predetermined range including a part or all of at least one edge of the captured image as the out-of-range object display area.

7. the virtual image for each object further includes a detailed information image of the object, and the position image and the detailed information image are linked to each other for each object; 7. The image processing method according to claim 5, wherein the image information output step provides a window portion within the captured image and superimposes the detailed information image linked to the position image of the object present in the captured image on the window portion.

8. a program for causing a computer to execute procedures including an imaging information acquisition procedure, a virtual information generation procedure, a map information management procedure, a user terminal information acquisition procedure, a specification procedure, and an image information output procedure; The imaging information acquisition step acquires an image captured by a user terminal in a specific area, The virtual information generation step generates a virtual image relating to the specific area; the map information management step includes managing position information of an object by linking it to a map of the specific area; the user terminal information acquisition step acquires location information and orientation information of the user terminal; the identification step includes identifying an object that is present in the captured image and an object that is not present in the captured image based on the map, position information of the user terminal, and orientation information of the user terminal, and, for an object that is not present in the captured image, identifying a direction in which the object is located relative to the user terminal based on the map, position information of the user terminal, and orientation information of the user terminal; the image information output step includes outputting a composite image in which the virtual image is superimposed on the captured image; the virtual image generated in the virtual information generation step includes a virtual image for each of the objects in the specific area, the virtual image for each of the objects includes a position image of the object, and the position images are linked to each other for each of the objects; The image information output step includes superimposing the position image on the object present in the captured image at a position corresponding to the object in the captured image; and providing an out-of-range object display unit in the captured image for the object that does not exist in the captured image, and outputting an image in which a part or all of a position image of the object is superimposed on the out-of-range object display unit according to a direction in which the object is located to the user terminal; the map information management step calculates a distance between the user terminal and the object on the map based on position information of the user terminal and position information of the object; The image information output step includes superimposing a part or all of the position images of the objects on the out-of-range object display unit so that the objects are arranged in one direction in ascending order of the distance.

Images