Display terminal, display method, and program

Abstract

To prevent the operation used for movement of a virtual viewpoint and the operation for scrolling from competing with each other.SOLUTION: A display terminal has a display control section that moves a virtual viewpoint for a wide-angle view image and changes a predetermined region image 711a on the basis of first operation performed within the predetermined region image, which is a predetermined region to be displayed of the wide-angle view image, and scrolls a plurality of images 711a and 712 including the predetermined region image on the basis of second operation performed within the predetermined region image.SELECTED DRAWING: Figure 20

Description

[Technical field]

[0001] The present disclosure relates to a display terminal, a display method, and a program. [Background technology]

[0002] Conventionally, when a display screen on a display is divided into multiple windows and images are displayed in the windows, each image is reduced in size so that it can fit in each window on the display screen (see Patent Document 1). In this case, for images that cannot fit in each window among the images displayed in each window on the display screen, the viewer moves a scroll bar or performs an operation such as dragging the image being displayed on the display screen to display other parts of the image displayed in the window.

[0003] On the other hand, when a wide-field image such as a spherical image is displayed, the image is curved and difficult for the viewer to see, so the viewer displays a predetermined area image, which is a predetermined area of ​​the wide-field image, on the display screen.Then, the viewer performs an operation such as dragging on the predetermined area image displayed on the display screen to move the virtual viewpoint in the wide-field image, thereby displaying another predetermined area image in the same wide-field image on the display screen. Summary of the Invention [Problem to be solved by the invention]

[0004] However, when a specified area image that is a specified area of ​​a wide-field image is included among the multiple images displayed in the display area, if the scroll target is a wide-field image, operations such as dragging will be used both for scrolling and for moving the virtual viewpoint, resulting in a problem of conflicting operations.

[0005] The present disclosure has been made in consideration of the above circumstances, and aims to prevent conflict between operations used to move the virtual viewpoint and operations used to scroll when a specified area image, which is a specified area of ​​a wide-field image, is included among multiple images displayed in the display area. [Means for solving the problem]

[0006] The disclosure of claim 1 is a display terminal having a display control unit that, based on a first operation performed within a predetermined area image, which is a predetermined area of ​​a wide-field-of-view image to be displayed, moves a virtual viewpoint with respect to the wide-field-of-view image to change the predetermined area image, and, based on a second operation performed within the predetermined area image, scrolls a plurality of images including the predetermined area image.

[0007] The disclosure of claim 16 is a display terminal that displays a plurality of images in a display area during a remote conference, the display terminal having a reception unit that receives scrolling of the display area including the plurality of images, and a display control unit that scrolls the plurality of images displayed in the display area upon reception of the scrolling by the reception unit, wherein a predetermined image among the plurality of images is a predetermined area image that shows a predetermined area of ​​the wide-field-of-view image, allowing a virtual viewpoint relative to the wide-field-of-view image to be moved. Effect of the Invention

[0008] As described above, according to the present disclosure, when a specified area image that is a specified area of ​​a wide-field image is included among multiple images displayed in the display area, it is possible to prevent conflict between the operation used to move the virtual viewpoint and the operation used for scrolling. [Brief description of the drawings]

[0009] [Figure 1] 1A is a left side view of the imaging device, FIG. 1B is a front view of the imaging device, and FIG. [Diagram 2] FIG. 13 is an image of the imaging device in use. [Diagram 3] (a) is a hemispherical image (front) taken by the photographing device, (b) is a hemispherical image (back) taken by the photographing device, and (c) is an image represented by the Mercator projection. [Figure 4] (a) A conceptual diagram showing how a sphere is covered with a Mercator image, and (b) a diagram showing a spherical image. [Diagram 5] FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is treated as a three-dimensional solid sphere. [Figure 6] (a) is a three-dimensional oblique view of Figure 5, (b) is a diagram showing the specified area image in the state of (a) displayed on a display, (c) is a diagram showing the specified area after the viewpoint of the virtual camera IC in (a) has been changed, and (d) is a diagram showing the specified area image in the state of (c) displayed on a display. [Figure 7] FIG. 1 illustrates points in three-dimensional Euclidean space in spherical coordinates. [Figure 8] FIG. 13 is a conceptual diagram showing the relationship between a predetermined area and a point of interest. [Figure 9] 1 is a schematic diagram of a communication system according to an embodiment of the present invention. [Figure 10] FIG. 2 is a diagram illustrating a hardware configuration of the imaging device. [Figure 11] FIG. 2 is a hardware configuration diagram of a relay device. [Figure 12] FIG. 2 is a hardware configuration diagram of a communication processing system and a communication terminal. [Figure 13] FIG. 2 is a diagram illustrating the functional configuration of the communication system. [Figure 14] FIG. 13 is a conceptual diagram of a user device management DB. [Figure 15] FIG. 13 is a conceptual diagram of a virtual room management DB. [Figure 16] FIG. 13 is a conceptual diagram of a viewpoint information management DB. [Figure 17] FIG. 2 is a sequence diagram showing communication processing of a wide-field image and each viewpoint information in the communication system. [Figure 18] 1 is a sequence diagram showing a communication process of normal images, screen sharing data, etc. in the communication system. [Figure 19] 13 is a flowchart showing a display process in response to mouse operations within and outside the predetermined area image when the communication terminal 9 is a PC. [Figure 20] FIG. 13 is a diagram showing an initial screen displayed by an application installed in the communication terminal 9. [Figure 21] A figure showing a state in which the display control unit changes the specified area image by moving the virtual viewpoint with respect to the wide-field image based on a drag operation performed within the specified area image of the initial screen displayed by the application. [Figure 22] FIG. 22 is a diagram showing a state in which, in the state shown in FIG. 21, the display control unit scrolls the predetermined area image based on a mouse wheel operation performed within the predetermined area image. [Figure 23] 13 shows an initial screen displayed by a Web browser installed in the communication terminal 9. FIG. [Figure 24] This figure shows the state in which the display control unit changes the specified area image by moving the virtual viewpoint with respect to the wide-field image based on a drag operation performed within the specified area image of the initial screen displayed by the web browser. [Diagram 25] FIG. 25 is a diagram showing a state in which, in the state shown in FIG. 24, the display control unit scrolls the predetermined area image based on a mouse wheel operation performed within the predetermined area image. [Figure 26] 13 is a flowchart showing a display process in response to an operation by an operator's finger within and outside the predetermined area image when the communication terminal 9 is a smartphone, a tablet terminal, or the like. [Figure 27] 13A and 13B are diagrams showing other examples of initial screens of the display terminal. [Figure 28] FIG. 13 is a diagram showing a state in which the display control unit replaces the predetermined area image with an image of the screen sharing data based on an operation of designating an arrow icon performed in a predetermined area image of an initial screen in another example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] [Overview of spherical images] A method for generating a spherical image will be described with reference to Fig. 1 to Fig. 8. A spherical image is also called a spherical panoramic image or a 360° panoramic image, and is an example of a wide-field video having a wide range of viewing angles. A wide-field image also includes a simple panoramic image of about 180°.

[0012] First, the appearance of the imaging device 10 will be described with reference to Fig. 1. The imaging device 10 is a digital camera for obtaining captured images that are the basis of a celestial sphere image. Fig. 1(a) is a left side view of the imaging device, Fig. 1(b) is a front view of the imaging device, and Fig. 1(c) is a plan view of the imaging device.

[0013] As shown in Fig. 1(a), the photographing device 10 is sized so that it can be held in one hand. Also, as shown in Figs. 1(a), (b), and (c), an image sensor 103a is provided on the front side (front side) and an image sensor 103b is provided on the back side (rear side) of the upper part of the photographing device 10. Also, as shown in Fig. 1(b), an operation unit 115 such as a shutter button is provided on the front side of the photographing device 10.

[0014] Next, the use of the photographing device 10 will be described with reference to FIG. 2. FIG. 2 is a conceptual diagram of the photographing device in use. As shown in FIG. 2, the photographing device 10 is communicably connected to a relay device 3 installed on some kind of stand, and is used to photograph surrounding subjects, scenery, and the like. In this case, two hemispherical images can be obtained by capturing images of subjects around the user using the image capturing element 103a and the image capturing element 103b shown in FIG. 1. If the omnidirectional image captured by the photographing device 10 is not transmitted to other communication terminals or systems, the relay device 3 is not necessary.

[0015] Next, an outline of a process for creating a celestial sphere image from an image captured by the image capture device 10 will be described with reference to Figs. 3 and 4. Fig. 3(a) shows a hemispherical image (front side) captured by the image capture device, Fig. 3(b) shows a hemispherical image (rear side) captured by the image capture device, and Fig. 3(c) shows an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). An image expressed by Mercator projection or the like (hereinafter referred to as "Mercator image") may also be used. Fig. 8(a) is a conceptual diagram showing a state in which a sphere is covered by an equirectangular projection image, and Fig. 8(b) shows a celestial sphere image. The "equirectangular projection image" is an equirectangular celestial sphere image as an example of the above-mentioned wide-field image.

[0016] As shown in Fig. 3(a), the image obtained by the image sensor 103a becomes a hemispherical image (front side) curved by a wide-angle lens 102a such as a fisheye lens described later. Also, as shown in Fig. 3(b), the image obtained by the image sensor 103b becomes a hemispherical image (rear side) curved by a wide-angle lens 102b such as a fisheye lens described later. Then, the photographing device 10 synthesizes the hemispherical image (front side) and the hemispherical image (rear side) flipped 180 degrees to create an equirectangular projection image EC as shown in Fig. 3(c).

[0017] Then, the photographing device 10 uses software such as OpenGL ES (Open Graphics Library for Embedded Systems) to paste an equirectangular projection image EC so as to cover the sphere as shown in FIG. 3(a), and creates a celestial sphere image CE as shown in FIG. 3(b). In this way, the celestial sphere image CE is expressed as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. OpenGL ES is merely an example of software that executes image processing, and the celestial sphere image CE may be created by other software. Also, the celestial sphere image CE may be a still image or a video. Note that, although an example in which the photographing device 10 generates a celestial sphere image has been described here, the communication processing system 5 or the communication terminals 7 and 9 may execute the same image processing or a part of the image processing steps.

[0018] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the Mercator image is pasted to cover the sphere as shown in FIG. 4(a), and a celestial sphere image as shown in FIG. 4(b) is created. In this way, the celestial sphere image is represented as an image in which the Mercator image faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. Note that the celestial sphere image may be a still image or a video.

[0019] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to people when they see it. Therefore, the communication terminals 7 and 9 can display a predetermined region of the spherical image (hereinafter, referred to as a "predetermined region image") as a planar image with little curvature, thereby enabling a display that does not give a sense of incongruity to people. This will be described with reference to Figs. 5 to 8.

[0020] Fig. 5 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional sphere. The virtual camera IC corresponds to the position of a virtual viewpoint of a user who views a celestial sphere image CE displayed as a three-dimensional sphere. In Fig. 6, (a) is a three-dimensional perspective view of Fig. 5, (b) is a diagram showing a predetermined area image in the state of (a) displayed on a display, (c) is a diagram showing a predetermined area after changing the viewpoint of the virtual camera IC in (a), and (d) is a diagram showing a predetermined area image in the state of (c) displayed on a display.

[0021] If the omnidirectional image CE generated in this manner is a stereoscopic sphere CS, the virtual camera IC is located inside the omnidirectional image CE as shown in Fig. 5. A predetermined area T in the omnidirectional image CE is an imaging area of ​​the virtual camera IC, and is specified by viewpoint information (also referred to as "predetermined area information") indicating the imaging direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the omnidirectional image CE.

[0022] Also, zooming of the predetermined area T can be expressed by moving the virtual camera IC closer to or farther away from the omnidirectional image CE. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE.

[0023] Furthermore, when the virtual viewpoint of the virtual camera IC is moved (also called "changed") from the state of Fig. 6(a) to the right (left when facing the drawing) as shown in Fig. 6(c), the predetermined area T in the omnidirectional image CE is accordingly moved to the predetermined area T', and the predetermined area image Q displayed on the predetermined display is changed to the predetermined area image Q'. As a result, the image shown in Fig. 6(b) is changed to the image shown in Fig. 6(d) and displayed on the display.

[0024] Next, the relationship between the viewpoint information and the image of the predetermined area T will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is a diagram showing points in a three-dimensional Euclidean space using spherical coordinates. Fig. 8 is a conceptual diagram showing the relationship between the predetermined area and a point of interest (center point).

[0025] Here, the coordinates of an arbitrary position when the center point CP shown in Fig. 7 is expressed in the spherical polar coordinate system are (r, θ, φ). (r, θ, φ) are the radius, polar angle, and azimuth angle, respectively. The radius r is the distance from the origin of the three-dimensional virtual space including the omnidirectional image to an arbitrary point (the center point CP in Fig. 8), and is equal to the distance f shown in Fig. 8.

[0026] Furthermore, as shown in FIG. 8, when the center of a predetermined area T, which is an imaging area of ​​a virtual camera IC, is considered to be the center point CP in FIG. 7, the trigonometric function shown in the following (Equation 1) generally holds. (L / f) = tan(α / 2) (Eq. 1) Here, f is the distance from the virtual camera IC to the center point CP. L is the distance between any vertex of the specified area T and the center point CP (2L is the diagonal). α is the angle of view. In this case, the viewpoint information for identifying the specified area T can be expressed by pan(θ), tilt(φ), and fov(α). Here, zooming of the specified area T can be expressed by widening or narrowing the range (arc) of the angle of view α.

[0027] [Outline of the communication system] Next, an overview of a communication system according to an embodiment of the present invention will be described with reference to Fig. 9. Fig. 9 is a schematic diagram of a communication system according to an embodiment of the present invention.

[0028] 9, the communication system 1 of this embodiment is composed of an imaging device 10, a relay device 3, and a communication terminal 7 and a communication terminal 9. The communication terminals 7 and 9 hold a remote conference via a communication network 100. The communication terminals 9 may be used at multiple locations. The communication terminals 7 and 9 may be referred to as "display terminals" that display images, etc.

[0029] Of these, the photographing device 10 is a digital camera for obtaining a spherical image as described above. The communication terminals 7 and 9 can hold a remote conference, so that a spherical image obtained by the photographing device 10 can be transmitted to the communication terminal 9 at a remote location that is another base. The relay device 3 is an example of a cradle that charges the photographing device 10 and transmits and receives data. The relay device 3 can perform data communication with the photographing device 10 via a contact point, and can also perform data communication with the communication processing system 5 via the communication network 100. The communication network 100 includes, for example, the Internet, a LAN (Local Area Network), a (wireless) router, and the like.

[0030] The communication processing system 5 is, for example, a computer server, and can perform data communication with the relay device 3 and the communication terminals 7 and 9 via the communication network 100.

[0031] The communication terminals 7 and 9 are, for example, a notebook PC (Personal Computer) or a smartphone, and can perform data communication with the communication processing system 5 via the communication network 100. OpenGL ES is installed in the communication terminals 7 and 9, and the communication terminals 7 and 9 create a predetermined area image (see FIG. 6) from the omnidirectional image received from the communication processing system 5. Note that the communication processing system 5 may be configured by a single computer server or may be configured by multiple computer servers.

[0032] Furthermore, the photographing device 10 and the relay device 3 are installed at predetermined positions by an observer X such as a worker at the construction site S. The communication terminal 7 is operated by the observer X. The communication terminal 9 is operated by a viewer A who is a construction manager (or construction supervisor) or the like in a remote location.

[0033] The communication processing system 5 transmits (distributes) the omnidirectional image obtained from the imaging device 10 via the relay device 3 to the communication terminals 7 and 9. Furthermore, the communication processing system 5 receives each piece of viewpoint information for identifying a predetermined area of ​​the predetermined area image being displayed (viewed by viewer A) on the communication terminal 9 from the communication terminal 9, and transmits each piece of viewpoint information to the communication terminal 7. Then, the communication terminal 7 displays a viewpoint display area based on each piece of viewpoint information in the predetermined area image (also referred to as "on the predetermined area image"), which is a predetermined area of ​​the omnidirectional image received from the communication processing system 5. This allows the observer X to know which predetermined area of ​​the omnidirectional image viewer A in a remote place is paying attention to when viewing it.

[0034] [Hardware configuration of the embodiment] Next, the hardware configurations of the image capturing device 10, relay device 3, and communication terminals 7 and 9 of this embodiment will be described in detail with reference to FIGS.

[0035] <Hardware configuration of the imaging device> Fig. 10 is a hardware configuration diagram of the photographing device. As shown in Fig. 10, the photographing device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121.

[0036] Of these, the imaging unit 101 is equipped with wide-angle lenses 102a, 102b (hereinafter, when there is no need to distinguish between them), each capable of capturing an image with an angle of view of 180° or more in order to form a hemispherical image, and two imaging elements 103a, 103b provided corresponding to each of the lenses 102a, 102b.

[0037] Furthermore, the imaging elements 103a, 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image by the lenses 102a, 102b, etc. into image data of an electrical signal and outputs it, a timing generation circuit that generates a horizontal or vertical synchronization signal or a pixel clock for this image sensor, and a group of registers in which various commands or parameters required for the operation of this imaging element are set, etc. Note that the configuration in which the imaging unit 101 has two wide-angle lenses is merely an example, and it may have only one lens, or three or more lenses.

[0038] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus).

[0039] The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to a CPU 111 via a bus 110. In addition, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, a network I / F 121, and the like.

[0040] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs a predetermined processing on each image data, and then synthesizes the image data to create data for an equirectangular projection image (an example of a wide-field image) described below.

[0041] The imaging control unit 105 generally sets commands and the like in the register groups of the imaging elements 103a and 103b using an I2C bus with the imaging control unit 105 as a master device and the imaging elements 103a and 103b as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

[0042] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function corresponding to video display on a display (for example, a display of an external terminal such as a smartphone that performs short-distance communication with the imaging device 10 using the short-distance communication circuit 117). In this case, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames / min).

[0043] As described later, the imaging control unit 105 also functions as a synchronization control means for synchronizing the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through an I / F bus and performs a predetermined process on the sound data.

[0044] The CPU 111 controls the overall operation of the photographing device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104, and data of a processed equirectangular projection image.

[0045] The operation unit 115 is a collective term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, etc. A user operates the operation unit 115 to input various imaging modes, imaging conditions, etc.

[0046] The input / output I / F 116 is a general term for an interface circuit (such as a USB I / F) with an external medium such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as necessary.

[0047] The short-range communication circuit 117 communicates with an external terminal (device) by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi via an antenna 117a provided in the photographing device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0048] The electronic compass 118 calculates the direction of the image capturing device 10 from the earth's magnetism and outputs the direction information. This direction information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The related information also includes each data of the image capture date and time and the data capacity of the image data.

[0049] The gyro sensor 119 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) accompanying the movement of the image capturing device 10. The change in angle is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image.

[0050] The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The image capturing device 10 calculates the attitude (angle with respect to the direction of gravity) of the image capturing device 10 itself (the image capturing device 10) based on the acceleration detected by the acceleration sensor 120. By providing the image capturing device 10 with the acceleration sensor 120, the accuracy of image correction is improved.

[0051] The network I / F 121 is an interface for performing data communication using a communication network 100 such as the Internet via a router or the like. The hardware configuration of the photographing device 10 is not limited to that shown here, and may be any configuration that can realize the functional configuration of the photographing device 10. At least a part of the above hardware configuration may be present on the relay device 3 or the communication network 100.

[0052] <Hardware configuration of relay device> Fig. 11 is a diagram showing a hardware configuration of the relay device 3. Note that Fig. 11 shows a hardware configuration when the relay device 3 is a cradle having a wireless communication function.

[0053] As shown in FIG. 11, the relay device 3 includes a CPU 310 , a ROM 302 , a RAM 303 , an EEPROM 304 , a CMOS sensor 305 , a bus line 310 , a communication unit 313 , an antenna 313 a , a GPS receiving unit 314 , and an input / output I / F 316 .

[0054] Of these, the CPU 301 controls the overall operation of the relay device 3. The ROM 302 stores programs such as an IPL (Initial Program Loader) used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301.

[0055] An EEPROM (Electrically Erasable and Programmable ROM) 304 reads and writes data under the control of the CPU 301. The EEPROM 304 stores an operating system (OS) executed by the CPU 301, other programs, and various data.

[0056] A CMOS (Complementary Metal Oxide Semiconductor) sensor 305 is a solid-state imaging element that captures an image of a subject and obtains image data under the control of the CPU 301.

[0057] The communication unit 313 uses an antenna 313a to communicate with the communication network 100 by wireless communication signals.

[0058] The GPS receiving unit 314 receives a GPS signal including position information (latitude, longitude, and altitude) of the relay device 3 by a Global Positioning Systems (GPS) satellite or an Indoor Messaging System (IMES) as an indoor GPS.

[0059] The input / output I / F 316 is an interface circuit (such as a USB I / F) electrically connected to the input / output I / F 116 of the photographing apparatus 10. The input / output I / F 316 may be wireless or wired.

[0060] The bus line 310 is an address bus, a data bus, etc. for electrically connecting the above-mentioned components.

[0061] <Hardware configuration of communication processing system and communication terminal> 12 shows the hardware configuration of the communication processing system 5. Note that the hardware configurations of the communication terminals 7 and 9 are similar to that of the communication processing system 5, and therefore will not be described.

[0062] As shown in FIG. 4, the communication processing system 5 is a computer and includes a CPU 501, a ROM 502, a RAM 503, an SSD 504, an external device connection I / F 505, a network I / F 506, a display 507, an operation unit 508, a media I / F 509, a bus line 510, a CMOS sensor 511, and a speaker 513.

[0063] Of these, the CPU 501 controls the overall operation of the communication processing system 5. The ROM 502 stores programs, such as an IPL, used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501.

[0064] The SSD 504 reads or writes various data under the control of the CPU 501. In addition, when the communication terminals 7 and 9 are smartphones or the like, the SSD 504 may not be provided. Also, instead of the SSD 504, a hard disk drive (HDD) may be provided. In the case of the communication terminals 7 and 9, an operating system (OS) is stored in the SSD 504, and a plurality of application software (hereinafter referred to as "applications") including functions for basic management and control of the communication terminals 7 and 9 and a web browser are stored. When an application is added, it is downloaded from the communication network 100 via the network I / F 506 and stored in the SSD 504. In this embodiment, the process executed by the communication terminals 7 and 9 is executed by the application or web browser of the communication terminals 7 and 9.

[0065] The external device connection I / F 505 is an interface for connecting various external devices, such as a display, a speaker, a keyboard, a mouse, a USB memory, and a printer.

[0066] The network I / F 506 is an interface for performing data communication via the communication network 100 .

[0067] The display 507 is a type of display unit such as a liquid crystal display or an organic EL (Electro Luminescence) display that displays various images.

[0068] An operation unit 508 is an input unit for selecting and executing various instructions such as various operation buttons, a power switch, a shutter button, and a touch panel, selecting a processing target, moving a cursor, and the like.

[0069] The media I / F 509 controls reading and writing (storing) of data from and to a recording medium 509m such as a flash memory. The recording medium 509m includes DVDs and Blu-ray Discs (registered trademarks). The media I / F 509 reads application software from the recording medium 509m. The application software read from the recording medium 509m by the media I / F 509 is stored in the SSD 504.

[0070] The CMOS sensor 511 is a type of imaging means that captures an image of a subject and obtains image data under the control of the CPU 501. A CCD sensor may be used instead of a CMOS sensor.

[0071] The microphone 512 collects sounds around the microphone 512 and converts them into sound data (sound information).

[0072] The speaker 513 is a circuit that converts an electrical signal into physical vibrations to produce sound such as music or voice.

[0073] The bus line 510 is an address bus, a data bus, or the like for electrically connecting the various components such as the CPU 501 shown in FIG.

[0074] [Functional configuration of the embodiment] Next, the functional configuration of this embodiment will be described with reference to FIGS.

[0075] <Functional configuration of the imaging device> 13, the photographing device 10 has a reception unit 12, a photographing unit 16, a sound collection unit 17, a connection unit 18, and a memory / readout unit 19. Each of these units is a function or means realized by any of the components shown in FIG. 10 operating in response to an instruction from the CPU 111 in accordance with the photographing / storage program expanded from the SRAM 113 onto the DRAM 114.

[0076] The image capturing device 10 also includes a storage unit 1000 that is configured by a ROM 112, an SRAM 113, and a DRAM 114 shown in FIG.

[0077] (Functional configuration of the imaging device) The reception unit 12 of the image capturing apparatus 10 is realized by the processing of the operation unit 115 for the CPU 111, and receives operation input from the user.

[0078] The photographing section 16 is realized mainly by processing of the imaging unit 101, image processing unit 104, and imaging control unit 105 by the CPU 111, and photographs scenery and the like to obtain a photographed image.

[0079] The sound collection section 17 is mainly realized by processing of the sound processing unit 109 from the CPU 111, and collects sounds around the image capturing device 10 and outputs sound data (sound information).

[0080] The connection unit 18 is realized mainly by processing from the CPU 111 to the input / output I / F 116, and performs data communication with the relay device 3.

[0081] The storage / reading unit 19 is realized mainly by the processing of the CPU 111 , and stores various data (or information) in the storage unit 1000 and reads various data (or information) from the storage unit 1000 .

[0082] <Functional configuration of relay device> 13, the relay device 3 has a communication unit 31 and a connection unit 38. Each of these units is a function or means realized by operating any of the components shown in FIG. 11 in response to an instruction from the CPU 301 in accordance with the program for the relay device 3 loaded from the EEPROM 304 onto the RAM 303.

[0083] (Functional configuration of relay device 3) The communication unit 31 of the relay device 3 is mainly realized by processing from the CPU 301 shown in FIG. 11 to the communication unit 313, and performs data communication between the image capturing device 10 and the communication processing system 5 via the communication network 100.

[0084] The connection unit 38 is mainly realized by processing from the CPU 301 to the input / output I / F 316, and performs data communication with the image capturing apparatus 10.

[0085] <Functional configuration of communication processing system> Next, each functional configuration of the communication processing system 5 will be described in detail with reference to Fig. 13. The communication processing system 5 has a communication unit 51, a reception unit 52, and a storage / readout unit 59. Each of these units is a function or means realized by any of the components shown in Fig. 12 operating in response to an instruction from the CPU 501 in accordance with the program for the communication processing system 5 loaded from the SSD 504 onto the RAM 503.

[0086] The communication processing system 5 also has a storage unit 5000 constructed by a RAM 503 and an SSD 504 shown in Fig. 12. In this storage unit 5000, a user device management DB 5001, a virtual room management DB 5002, and a viewpoint information management DB 5003 are constructed.

[0087] (User / device management DB) 14 is a conceptual diagram of a user / device management DB. The user / device management DB 5001 is configured in a table format, and stores and manages user IDs (or device IDs), names, and IP addresses in association with each other.

[0088] Among these, the user ID is an example of user identification information for identifying a user (observer X, viewer A). The device ID is an example of device identification information for identifying a device such as the image capture device 10. In addition, in FIG. 9, when a head mounted display or the like is used in addition to the image capture device 10, the head mounted display or the like is also treated as a device.

[0089] The name is the name of the user or device.

[0090] The IP address is an example of destination identification information of a communication terminal used by a user, a device such as the photographing apparatus 10, or the like.

[0091] (Virtual room management DB) 15 is a conceptual diagram of the virtual room management DB 5002. The virtual room management DB 5002 is configured in a table format, and stores and manages a virtual room ID, a virtual room name, a device ID, an observer ID, a viewer ID, and storage (information on a storage location of image data) in association with each other.

[0092] Of these, the virtual room ID is an example of virtual room identification information for identifying a virtual room.

[0093] The virtual room name is the name of the virtual room and is given by the user or the like.

[0094] The device ID is the same as the device ID in FIG. 14, and is the ID of the device that has participated in the virtual room indicated by the virtual room ID of the same record.

[0095] The observer ID is an example of observer identification information for identifying an observer ID in particular among the user IDs in FIG. 14, and is the ID of an observer who has participated in the virtual room indicated by the virtual room ID of the same record.

[0096] The viewer ID is an example of viewer identification information for particularly identifying a viewer ID among the user IDs in FIG. 14, and is the ID of a viewer who has participated in the virtual room indicated by the virtual room ID of the same record.

[0097] The storage is an example of storage location information indicating the location where the wide-field image or the like is stored, and specifically indicates a URL, a file path, or the like.

[0098] (Viewpoint information management DB) 16 is a conceptual diagram of the viewpoint information management DB 5003. The viewpoint information management DB 5003 is configured in a table format, and stores and manages a viewer ID, an IP address, viewpoint information (pan, tilt, fov), and a time stamp in association with each other.

[0099] Among these, the viewer ID is the same as the viewer ID in FIG.

[0100] The IP addresses are the same as those in FIG.

[0101] The viewpoint information (pan, tilt, fov) is viewpoint information sent from the communication terminal of the viewer indicated by the viewer ID of the same record.

[0102] The timestamp indicates the time when the viewpoint information of the same record was sent.

[0103] (Functional configuration of the communication processing system) Next, each functional configuration of the communication processing system 5 will be described in detail with reference to FIG.

[0104] The communication unit 51 of the communication processing system 5 is mainly realized by processing from the CPU 501 shown in FIG. 12 to the network I / F 505, and performs data communication with other devices (relay device 3, communication terminals 7 and 9) via the communication network 100.

[0105] The reception unit 52 is realized by the processing of the operation unit 508 for the CPU 501, and receives operation input from a user (here, a system administrator or the like).

[0106] The storage / reading unit 59 is realized mainly by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000 .

[0107] <Functional configuration of communication terminal 7> Next, the functional configuration of the communication terminal 7 will be described in detail with reference to Fig. 13. The communication terminal 7 has a communication unit 71, a reception unit 72, a display control unit 74, a sound input / output control unit 75, an image capture unit 76, a sound collection unit 77, and a storage / readout unit 79. Each of these units is a function or means realized by operating any of the components shown in Fig. 12 by an instruction from the CPU 501 in accordance with the program for the communication terminal 7 loaded from the SSD 504 onto the RAM 503.

[0108] The communication terminal 7 also has a storage unit 7000 constructed by the RAM 503 and SSD 504 shown in Fig. 12. In addition to the multiple application software described in Fig. 12, a viewpoint information management DB 7003 is also constructed in this storage unit 7000. This viewpoint information management DB 7003 stores and manages one record of viewpoint information etc. sent from the communication processing system 5, and has the same configuration as the viewpoint information management DB 7003, so a description thereof will be omitted.

[0109] (Functional configuration of communication terminal 7) Next, each functional configuration of the communication terminal 7 will be described in detail with reference to FIG.

[0110] The communication unit 71 of the communication terminal 7 is mainly realized by processing from the CPU 501 shown in FIG. 11 to the network I / F 505, and performs data communication with other devices (communication processing system 5) via the communication network 100.

[0111] The reception unit 72 is realized by the processing of the operation unit 508 for the CPU 501, and receives operation input from a user (here, the observer X).

[0112] The display control unit 74 is realized by the processing of the CPU 501, and performs control for displaying various images on the display 507 of the communication terminal 7 or an external display connected to the external device connection I / F 505.

[0113] The sound input / output control unit 75 is realized by the processing of the CPU 501, and performs control for collecting sound from the microphone 512 of the communication terminal 7 or an external microphone connected to the external device connection I / F 505. The sound input / output control unit 75 also performs control for outputting sound from the speaker 513 of the communication terminal 7 or an external speaker connected to the external device connection I / F 505.

[0114] The photographing unit 76 is realized by processing on the CMOS 511 of the communication terminal 7 from the CPU 501 of the communication terminal 7 and image processing by the CPU 501 of the communication terminal 7, and photographs the observer X, scenery, etc., to obtain a photographed image.

[0115] The sound collection unit 77 is realized by processing of the microphone 512 of the communication terminal 7 by the CPU 501 of the communication terminal 7, and sound (audio) processing by the CPU 501 of the communication terminal 7, and collects sounds around the communication terminal 7 and outputs sound data (sound information).

[0116] The storage / reading unit 79 is realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 7000 and reads various data (or information) from the storage unit 7000 .

[0117] <Functional configuration of communication terminal 9> Next, each functional configuration of the communication terminal 9 will be described in detail with reference to Fig. 13. The communication terminal 9 has a communication unit 91, a reception unit 92, a display control unit 94, a sound input / output control unit 95, an image capture unit 96, and a storage / readout unit 99. Each of these units is a function or means realized by operating any of the components shown in Fig. 12 by an instruction from the CPU 501 in accordance with the program for the communication terminal 9 loaded from the SSD 504 onto the RAM 503.

[0118] The communication terminal 9 also includes a storage unit 9000 constructed by the RAM 503 and the SSD 504 shown in FIG.

[0119] The communication unit 91 of the communication terminal 9 is realized by processing from the CPU 501 shown in FIG. 12 to the network I / F 505, and performs data communication with other devices (communication processing system 5) via the communication network 100.

[0120] The reception unit 92 is realized by the processing of the operation unit 508 for the CPU 501, and receives operation input from a user (viewer A in this case).

[0121] The display control unit 94 is realized by the processing of the CPU 501, and performs control for displaying various images on the display 507 of the communication terminal 9 or an external display connected to the external device connection I / F 505.

[0122] The sound input / output control unit 95 is realized by the processing of the CPU 501, and performs control for outputting sound to the speaker 513 of the communication terminal 9 or an external speaker connected to the external device connection I / F 505.

[0123] The photographing unit 96 is realized by processing on the CMOS 511 of the communication terminal 9 from the CPU 501 of the communication terminal 9 and image processing by the CPU 501 of the communication terminal 9, and photographs the viewer A, scenery, etc. to obtain a photographed image.

[0124] The sound collection unit 97 is realized by processing of the microphone 512 of the communication terminal 9 by the CPU 501 of the communication terminal 9, and sound (audio) processing by the CPU 501 of the communication terminal 9, and collects sounds around the communication terminal 9 and outputs sound data (sound information).

[0125] The storage / reading unit 99 is realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 7000 and reads various data (or information) from the storage unit 7000 .

[0126] [Processing or Operation of the Embodiment] Next, the processing or operation of this embodiment will be described with reference to Fig. 17 to Fig. 28. Note that the following processing is processing performed after the image capturing device 10 and the communication terminals 7 and 9 have already participated in the same virtual room.

[0127] <Transmission and reception processing of wide-field images and sound information in communication systems> First, the transmission and reception process of the wide-field image, sound information, and the like in the communication system 1 will be described with reference to Fig. 17 and Fig. 18. In describing the process in Fig. 17 and Fig. 18, the virtual room ID, observer ID, and viewer ID shown in Fig. 15 are shown as "001r", "100x", and "100a", respectively, but these are merely examples and are not limited to these.

[0128] (Transmission and reception of wide-field image and sound information) First, the transmission and reception process of wide-field-of-view images, sound information, etc. in the communication system will be described with reference to Fig. 17. Fig. 17 is a sequence diagram showing the communication process of wide-field-of-view images, each viewpoint information, etc. in the communication system. Note that the processes S11 to S16 in Fig. 17 are repeated, for example, 30 or 60 times per second.

[0129] S11: In the photographing device 10, the photographing unit 16 photographs the inside of the construction site S in a 360-degree view to obtain a wide-field-of-view image, and then the connection unit 18 sends the wide-field-of-view image to the relay device 3. At the same time, in the photographing device 10, the sound collection unit 17 collects sounds in the construction site S to obtain sound information, and then the connection unit 18 sends the sound information to the relay device 3. In this case, the connection unit 18 also transmits a virtual room ID for identifying the virtual room in which the photographing device 10 is participating and a device ID for identifying the photographing device 10. As a result, the connection unit 38 of the relay device 3 obtains the wide-field-of-view image, sound information, virtual room ID, and device ID.

[0130] S12: In the relay device 3, the communication unit 31 transmits each piece of information (wide-field image, sound information, virtual room ID, and device ID) acquired by the connection unit 38 in process S11 to the communication processing system 5 via the communication network 100. As a result, in the communication processing system 5, the communication unit 51 receives each piece of information (wide-field image, sound information, virtual room, and device ID).

[0131] S13: In the communication processing system 5, the storage / readout unit 59 searches the virtual room management DB 5002 based on the virtual room ID "001r" received in process S12 to read out the observer ID "100x" which is the user ID of observer X and the viewer ID "100a" which is the user ID of viewer A who participate in the same virtual room as the image capturing device 10. The storage / readout unit 59 also searches the user / device management DB 5001 based on the observer ID and viewer ID read out to read out the corresponding IP address of the communication terminal 7 of observer X and the IP address of the communication terminal 9 of viewer A. Then, the communication unit 51 refers to the IP address of the communication terminal 7 and transmits the wide-field image and sound information received in process S12 to the communication terminal 7. As a result, the communication unit 71 of the communication terminal 7 receives the wide-field image and sound information.

[0132] S14: The communication unit 51 of the communication processing system 5 refers to the IP address of the communication terminal 9, and transmits the wide-field image and sound information received in process S12 to the communication terminal 9. As a result, the communication unit 91 of the communication terminal 9 receives the wide-field image and sound information.

[0133] (Transmitting and receiving viewpoint information) S15: Next, in the communication terminal 9, the display control unit 94 displays a predetermined area image showing a predetermined area in the wide-field image received in process S14, and the sound input / output control unit 95 outputs sound based on the sound information received in process S14. In addition, the reception unit 92 receives a screen operation by the viewer A, and the display control unit 94 changes the predetermined area T (see FIG. 6(a)) and displays a predetermined area image showing a predetermined area T' (see FIG. 6(c)) in which objects such as products and buildings that interest the viewer A are displayed. Then, the communication unit 91 of the communication terminal 9 transmits viewpoint information for identifying the predetermined area T' in the wide-field image to the communication processing system 5. In this case, the viewpoint information includes a viewer ID for identifying the viewer A who is the sender.

[0134] As a result, the communication unit 51 of the communication processing system 5 receives the viewpoint information. Furthermore, the storage / reading unit 59 associates the viewpoint information (pan, tilt, fov) received in processing S15 with the viewer ID and stores them as one record in the viewpoint information management DB 5003. In this case, the storage / reading unit 59 also associates and stores, as the same record, time information (timestamp) indicating the time when the viewpoint information was received in processing S17 and the viewer ID of viewer A, which is the user ID managed in the user / device management DB 5001.

[0135] S16: The storage / readout unit 59 of the communication processing system 5 reads out the viewpoint information, etc., which is one record's worth stored in the viewpoint information management DB 5003 in the process S15, and the communication unit 51 transmits the viewpoint information, etc. to the communication terminal 7. As a result, the communication unit 71 of the communication terminal 7 receives the viewpoint information, etc. In addition, the communication processing system 5 searches the virtual room management DB 5002 shown in FIG. 15 based on the virtual room ID "001r" identified in the process S12, and reads out the corresponding observer ID "100x". Furthermore, the user / device management DB 5001 shown in FIG. 14 is searched based on this observer ID, and the corresponding IP address is read out. As a result, the communication unit 51 transmits the viewpoint information, etc. to the communication terminal 7 of only the observer X out of the observer X and the viewer A.

[0136] (Normal image and sound information transmission and reception) FIG. 18 is a sequence diagram showing communication processing of normal images, screen sharing data, and the like in the communication system.

[0137] S21: In the communication terminal 7, the photographing unit 76 photographs the observer X, etc. to obtain a planar normal image, and then the communication unit 71 transmits the normal image to the communication processing system 5. At the same time, in the communication terminal 7, the sound collecting unit 77 collects sounds around the communication terminal 7 to obtain sound information, and then the communication unit 71 transmits the sound information to the communication processing system 5. In this case, the communication unit 71 also transmits a virtual room ID for identifying the virtual room in which the communication terminal 7 is participating and a user ID for identifying the observer X. As a result, the communication unit 51 of the communication processing system 5 receives the normal image, sound information, virtual room ID, and user ID.

[0138] S22: In the communication processing system 5, the storage / reading unit 59 searches the virtual room management DB 5002 shown in FIG. 15 based on the virtual room ID received in the process S21 to read out the viewer ID "100a" which is the user ID of the viewer A participating in the same virtual room as the viewer X. Also, based on the viewer ID read out by the storage / reading unit 59, the storage / reading unit 59 searches the user / device management DB 5001 shown in FIG. 14 to read out the IP address of the corresponding communication terminal 9 of the viewer A. Then, the communication unit 51 refers to the IP address of the communication terminal 9 and transmits the normal image and sound information received in the process S21 to the communication terminal 9. As a result, the communication unit 91 of the communication terminal 9 receives the normal image and sound information. Then, the display control unit 94 causes the display 507 of the communication terminal 9 to display the image in which the viewer X appears. Also, the sound input / output control unit 95 causes the speaker 513 of the communication terminal 9 to output the sound generated from the sound information.

[0139] S23: In the communication terminal 9, the photographing unit 96 photographs the viewer A, etc. to obtain a flat normal image, and then the communication unit 91 transmits the normal image to the communication processing system 5. At the same time, in the communication terminal 9, the sound collection unit 97 collects sounds around the communication terminal 9 to obtain sound information, and then the communication unit 91 transmits the sound information to the communication processing system 5. In this case, the communication unit 91 also transmits a virtual room ID for identifying the virtual room in which the communication terminal 9 is participating and a user ID for identifying the viewer A. As a result, the communication unit 51 of the communication processing system 5 receives the normal image, sound information, virtual room ID, and user ID.

[0140] S24: In the communication processing system 5, the storage / reading unit 59 searches the virtual room management DB 5002 shown in FIG. 15 based on the virtual room ID "001r" received in the process S23, thereby reading out the observer ID "100x" which is the user ID of the observer X participating in the same virtual room as the viewer A. Also, based on the observer ID read out by the storage / reading unit 59, the user / device management DB 5001 shown in FIG. 14 is searched to read out the IP address of the communication terminal 7 of the corresponding observer X. Then, the communication unit 51 refers to the IP address of the communication terminal 7, and transmits the normal image and sound information received in the process S23 to the communication terminal 7. As a result, the communication unit 71 of the communication terminal 7 receives the normal image and sound information. Then, the display control unit 74 causes the display 507 of the communication terminal 7 to display the image in which the viewer A is reflected. Also, the sound input / output control unit 75 causes the speaker 513 of the communication terminal 7 to output the sound generated from the sound information.

[0141] (Sending and receiving screen sharing data) S25: In the communication terminal 9, the display control unit 94 further displays an image related to the screen sharing data (an image different from the normal image) on the display 507 of the communication terminal 9. Furthermore, the communication unit 91 of the communication terminal 9 transmits the screen sharing data to the communication processing system 5. In this case, the communication unit 91 also transmits to the communication processing system 5 a virtual room ID for identifying the virtual room in which the communication terminal 9 is participating and a user ID for identifying the viewer A. As a result, the communication unit 51 of the communication processing system 5 receives the screen sharing data, the virtual room ID, and the user ID from the communication terminal 9.

[0142] S26: In the communication processing system 5, the storage / reading unit 59 searches the virtual room management DB 5002 shown in FIG. 15 based on the virtual room ID "001r" received in the process S25, thereby reading out the observer ID "100x" which is the user ID of the observer X participating in the same virtual room as the viewer A. Also, based on the observer ID read out by the storage / reading unit 59, the user / device management DB 5001 shown in FIG. 14 is searched out to read out the IP address of the corresponding communication terminal 7 of the observer X. Then, the communication unit 51 refers to the IP address of the communication terminal 7, and transmits the normal image and sound information received in the process S25 to the communication terminal 7. As a result, the communication unit 71 of the communication terminal 7 receives the screen sharing data. Then, the display control unit 74 causes the display 507 of the communication terminal 7 to display an image related to the screen sharing data.

[0143] <Display process in communication terminal 9> (When communication terminal 9 is a PC) Next, the display process in the communication terminal 9 will be described with reference to FIGS.

[0144] Fig. 19 is a flowchart showing a display process in response to mouse operations within and outside the predetermined area image when the communication terminal 9 is a PC. Fig. 20 is a diagram showing an initial screen displayed by an application installed in the communication terminal 9.

[0145] Here, a case will be described in which a viewer A as a user uses a mouse to operate a cursor c1 on a screen 700 displayed by an application on a display 507 of a communication terminal 9 to perform designation, selection, or the like. The mouse is an example of an operation device, and operation devices also include a pointing device, a touch pad of a notebook PC, and the like. The operation device may be provided in the communication terminal 9, or may be an external device capable of wired or wireless communication with the external device connection I / F 506 of the communication terminal 9.

[0146] First, the display control unit 94 causes the display 507 of the communication terminal 9 to display a screen 700 as shown in FIG. 20 as an initial screen through the processes in FIGS.

[0147] Screen 700 includes a main display area 710 with a large display area on the left side, and a sub-display area 720 with a small display area on the right side. For example, the width of main display area 710 is about 70% of the width of screen 700, and the width of sub-display area 720 is about 20% of the width of screen 700. The remaining 10% is scroll bar b3 and a margin. Note that main display area 710 and sub-display area 720 may each be separate screens. Note that scrolling refers to displaying text or charts on a computer screen by moving them up and down or left and right, as if reading them.

[0148] In the main display area 710, a predetermined area image 711a is displayed, which is a predetermined area (display area) displayed by the display control unit 94 among the wide-field image sent from the imaging device 10 via the communication processing system 5. A mark m1 indicating that the predetermined area image 711a can move the virtual viewpoint is displayed in the lower right part of the predetermined area image 711a. Furthermore, an enlargement button b1, which is pressed when the predetermined area image 711a is enlarged, and a reduction button b2, which is pressed when the predetermined area image 711a is enlarged, are displayed in the lower right part of the predetermined area image 711a. The enlargement button b1 and the reduction button b2 are examples of a reception area.

[0149] Furthermore, an image 712 related to the screen sharing data is displayed below the predetermined area image 711a in the main display area 710. Here, due to the display area of ​​the main display area 710, only a part of the image 712 is displayed.

[0150] Further, a scroll bar b3 for scrolling images in the main display area 710 is displayed so as to move vertically on the right side of the main display area 710. When the viewer A moves the scroll bar b3, the display control unit 94 moves (scrolls) a plurality of images including the predetermined area image 711a in the main display area 710 (the predetermined area image 711a and the image 712 related to the screen sharing data, etc.).

[0151] The display control unit 94 may display the predetermined area image 711a and the image 712 related to the screen sharing data side by side instead of vertically. In this case, the scroll bar b is displayed so as to move in the horizontal direction.

[0152] Furthermore, the sub-display area 720 displays a normal image 721 of viewer A captured by the photographing unit 96 of the communication terminal 9, and a normal image 722 of observer X sent from the communication terminal 7 via the communication processing system 5 by processes S21 and S22.

[0153] In the above state, the display processing performed by the display control unit 94 in response to the processing content on the screen 700 by the viewer A using the mouse will be described with reference to FIG.

[0154] S31: The reception unit 92 judges whether the mouse operation by the viewer A is an operation by the cursor c1 within the predetermined region image.

[0155] S32: When the reception unit 92 determines in the process S31 that the operation is within the predetermined region image (S31; YES), the reception unit 92 determines whether the operation is for the enlargement icon b1 or the reduction icon b2.

[0156] S33: If the reception unit 92 determines that the operation is not performed on the enlarged icon b1 or the reduced icon b2 (S32; NO), the reception unit 92 determines whether the mouse operation method is a drag or a mouse wheel operation. In this case, the drag and mouse wheel operations are performed with the same mouse. Note that if two operation devices such as mice are used simultaneously, and the reception unit 92 can distinguish and recognize each operation device, the drag of one operation device may be recognized as a drag, and the drag of the other operation device may be recognized as the same operation as the mouse wheel.

[0157] S34: When the reception unit 92 determines that the operation is a drag (S33; drag), the display control unit 94 changes the predetermined area image in accordance with a change in the virtual viewpoint for the wide-field-of-view image. For example, in FIG. 20, when the viewer A uses the mouse to move the cursor c1 to the position of the cursor c1' on the predetermined area image 711a, the reception unit 92 detects the movement of the cursor c1, and the display control unit 94 changes the predetermined area image 711a to the predetermined area image 711b as shown in FIG. 21. The predetermined area image 711b is an image of the upper side of the predetermined area image 711a (the side closer to the north pole of the wide-field-of-view image) in the same wide-field-of-view image.

[0158] S35: When the reception unit 92 determines that the operation is a mouse wheel operation (S33; mouse wheel operation), the display control unit 94 scrolls the images (predetermined area image 711b and image 712 of screen sharing data) in the main display area 710. For example, in FIG. 21, when the viewer A uses the mouse to perform an operation of rotating the mouse wheel, the reception unit 92 detects the rotation of the mouse wheel, and the display control unit 94 scrolls the predetermined area image 711b and image 712 related to the screen sharing data in the main display area 710 as shown in FIG. 22. In this case, the display control unit 94 moves the scroll bar b3 according to the rotation of the mouse wheel. The position of the scroll bar b3 after the movement is the position where the display control unit 94 scrolls the predetermined area image 711b and image 712 related to the screen sharing data when the viewer A moves the scroll bar b3, as shown in FIG. 22.

[0159] In addition, when only the specified area image 711a is displayed in the display area 710, the display control unit 94 may move (scroll) at least the specified area image 711a without moving (scrolling) the images in the sub-display area 720 (the normal image 721 of viewer A, the normal image 722 of observer X).

[0160] S36: In addition, in the above process S32, if the reception unit 92 determines that the operation is for the enlargement icon b1 or reduction icon b2 (S32; YES), the display control unit 94 enlarges and displays the predetermined area image when the enlargement icon b1 is pressed, and reduces and displays the predetermined area image when the reduction icon b2 is pressed. That is, the display control unit 94 executes a predetermined process (e.g., enlargement display, reduction display) by prioritizing the pressing of the enlargement icon b1 and reduction icon b2 in the predetermined area image over mouse operations such as dragging within the predetermined area image and mouse wheel operation.

[0161] S37: In addition, in the above process S31, if the reception unit 92 determines that the operation is not within the predetermined area image (S31; NO), the display control unit 94 scrolls the predetermined area image, etc., according to the operation content. For example, in Fig. 21, if the viewer A performs an operation to move the cursor c1 on the scroll bar b3 by left-clicking the mouse, the reception unit 92 detects the movement of the scroll bar b3, and the display control unit 94 scrolls the images (the predetermined area image 711b and the image 712 related to the screen sharing data) within the main display area 710 as shown in Fig. 22.

[0162] 20 to 22 show screens displayed by an application installed in the communication terminal 9, but as shown in Fig. 23 to 25, the screens may be displayed by a web browser installed in the communication terminal 9. Fig. 23 to Fig. 25 correspond to Fig. 20 to Fig. 22, respectively.

[0163] Furthermore, in either the application or the Web browser, the display control units 74, 94 execute the processes S31 to S37 (and the processes S51 to S56 described below).

[0164] 23 to 25, parts denoted by the same reference numerals as those in FIGS. 20 to 22 play the same role. Furthermore, in FIGS. 23 to 25, a scroll bar b4 for scrolling the image in the sub display area 720 is displayed to move vertically on the right side of the sub display area 720. When the scroll bar b4 is moved by the viewer A, the display control unit 94 moves the images in the sub display area 720 (the normal image 721 of the viewer A and the normal image 722 of the observer X). When the scroll bar b4 is moved, the display control unit 94 may move the image in the main display area 710 and the image in the sub display area 720. The scroll bar b4 may also be displayed on the screens shown in FIGS. 20 to 22.

[0165] (When the communication terminal 9 is a smartphone, tablet terminal, etc.) Next, the display process in the communication terminal 9 will be described with reference to Fig. 26. If the communication terminal 9 is a smartphone, tablet terminal, or the like, the display size is smaller than that of a PC, so the enlarge button b1 and reduce button b2 are not displayed. However, it is possible to enlarge an image by pinching out, and to reduce an image by pinching in.

[0166] 26 is a flowchart showing a display process in response to an operation by an operator's finger within and outside the predetermined area image when the communication terminal 9 is a smartphone, tablet terminal, etc. A touch panel for detecting contact with a human body part such as a finger is provided on the smartphone or tablet terminal, and the reception unit 92 receives various operations by the user's contact with the touch panel. Also, when the communication terminal 9 is a smartphone, tablet terminal, etc. with a relatively small display, the display control unit 94 does not need to display the sub display area 720 shown in FIG. 23.

[0167] S51: The reception unit 92 judges whether the operation with the finger of the viewer A is an operation within the predetermined area image.

[0168] S52: If the reception unit 92 determines in process S51 that the operation is within a specified area image (S51; YES), the reception unit 92 determines whether the finger operation method is a one-finger swipe, a two-finger swipe, or a pinch out or pinch in using two fingers.

[0169] S53: When the reception unit 92 determines that the swipe is with one finger (S53; swipe with one finger), the display control unit 94 changes the predetermined area image according to the change of the virtual viewpoint for the wide-field image. For example, in FIG. 20 or FIG. 23, when the viewer A touches the predetermined area image 711a with one finger and swipes the one finger downward in the same manner as the movement from the cursor c1 to the cursor c1', the reception unit 92 detects the movement of the one finger, and the display control unit 94 changes the predetermined area image 711a to the predetermined area image 711b as shown in FIG. 21 or FIG. 24. The predetermined area image 711b is an image of the upper side of the predetermined area image 711a in the same wide-field image (the side closer to the north pole of the wide-field image).

[0170] S54: When the reception unit 92 determines that the operation is a swipe operation with two fingers (S52; swipe with two fingers), the display control unit 94 scrolls the images (predetermined area image and image of screen sharing data) in the main display area 710. For example, in FIG. 21 or FIG. 24, when the viewer A touches two fingers on the predetermined area image 711b and swipes the two fingers upward, the reception unit 92 detects the movement of the two fingers, and the display control unit 94 scrolls the predetermined area image 711b and the image 712 related to the screen sharing data in the main display area 710 as shown in FIG. 22 or FIG. 25. In this case, the display control unit 94 moves the scroll bar b3 according to the rotation of the mouse wheel with two fingers. The position of the scroll bar b3 after the movement is the position where the display control unit 94 scrolls the predetermined area image 711b and the image 712 when the viewer A moves the scroll bar b3, as shown in FIG. 22 or FIG. 25.

[0171] S55: Also, if the reception unit 92 determines that the operation is a pinch out or pinch in operation with two fingers on the touch panel (S52; pinch out or pinch in), the display control unit 94 enlarges the image of the specified area in the case of a pinch out, or reduces the image of the specified area in the case of a pinch in.

[0172] S56: In addition, in the above process S51, if the reception unit 92 determines that the operation is not within the predetermined area image (S51; NO), the display control unit 94 scrolls the predetermined area image, etc., according to the operation content. For example, in FIG. 21 or FIG. 24, if the viewer A touches the scroll bar b3 with one finger and swipes downward, the reception unit 92 detects the movement of the scroll bar b3, and the display control unit 94 scrolls the predetermined area image 711b and the image 712 related to the screen sharing data within the main display area 710, as shown in FIG. 22 or FIG. 25.

[0173] <Other display processes in the communication terminal 9> Next, other display processes in the communication terminal 9 will be described with reference to FIGS.

[0174] Fig. 27 is a diagram showing another example of an initial screen of the display terminal. Fig. 28 is a diagram showing a state in which the display control unit replaces the image of the predetermined area with the image of the screen sharing data based on an operation of designating an arrow icon performed in the image of the predetermined area of ​​the initial screen of the other example.

[0175] First, the display control unit 94 causes the display 507 of the communication terminal 9 to display a screen 800 as shown in FIG. 27 as an initial screen through the processes in FIGS.

[0176] A screen 800 includes a main display area 810 having a large display area on the left side, and a sub-display area 820 having a small display area on the right side. Note that the main display area 810 and the sub-display area 820 may each be separate screens.

[0177] In the main display area 810, a predetermined area image 811a is displayed, which is a predetermined area (display area) displayed by the display control unit 94 out of the wide-field image sent from the imaging device 10 via the communication processing system 5. In the lower right part of the predetermined area image 811a, a mark m1 is displayed indicating that the predetermined area image 811a can move the virtual viewpoint. That is, the predetermined area image 811a can move the virtual viewpoint in the same way as the predetermined area image 711a in FIG. 20.

[0178] Furthermore, in the lower right corner of the predetermined area image 811a, there are displayed an enlargement button b1 to be pressed when enlarging the predetermined area image 811a, and a reduction button b2 to be pressed when enlarging the predetermined area image 811a. That is, the predetermined area image 811a can be enlarged or reduced in size, similar to the predetermined area image 711a in FIG.

[0179] 27, unlike the main display area 710 shown in Fig. 20, no image related to the screen sharing data is displayed below the predetermined area image 811a in the main display area 810. Also, unlike the screen 700 shown in Fig. 20, no scroll bar for scrolling the images in the main display area 810 is displayed on the right side of the main display area 810.

[0180] In addition, the sub-display area 820 displays, from top to bottom, a normal image 821 of viewer A captured by the imaging unit 96 of the communication terminal 9, and a normal image 822 of observer X sent from the communication terminal 7 via the communication processing system 5 by processes S21 and S22.

[0181] 22. Furthermore, unlike Fig. 20, a thumbnail 823 of an image related to the screen sharing data is displayed at the bottom of the sub display area 820. The thumbnail 823 corresponds to the thumbnail of the image 712 shown in Fig. 22.

[0182] 27, an arrow button b5 pointing to the left and an arrow button b6 pointing to the right are displayed on the predetermined area image 811a. The arrow button b5 is a button for cyclically displaying each image from the sub display area 820 to the main display area 810 in the following order: the predetermined area image 811a, a thumbnail 823 of an image related to the screen sharing data, a normal image 822 of the observer X, and a normal image 821 which is a self-portrait of the viewer A.

[0183] For example, when the reception unit 92 detects that the arrow button b5 has been pressed by the viewer A, the display control unit 94 cyclically replaces the predetermined area image 811a in the main display area 810 with the image (thumbnail 823) related to the screen sharing data in the sub display area 820, as shown in Fig. 28. Note that when the thumbnail 823 is moved from the sub display area 820 to the main display area 810, it is displayed as an image 812 as shown in Fig. 27, rather than as a thumbnail.

[0184] On the other hand, the arrow button b6 is a button for cyclically displaying each image in the reverse order to that of the arrow button b5, that is, in the order of the predetermined area image 811a, the normal image 821 which is the self-portrait of the viewer A, the normal image 822 of the observer X, and the thumbnail 823 of the image related to the screen sharing data. Note that the cyclical direction of each image may be reversed between the arrow button b5 and the arrow button b6.

[0185] In this way, when the reception unit 92 detects that the arrow button b5 or the arrow button b6 has been pressed, the display control unit 94 can switch the predetermined area image 811a in the main display area 810 with the image in the sub display area 820. The arrow button b5 and the arrow button b6 are examples of a reception area.

[0186] [Main Effects of the Embodiments] (1) As described above, according to this embodiment, the communication terminal 9 changes the predetermined area image, which is a predetermined area to be displayed in the wide-field-of-view image, in response to the operation of the viewer A on the wide-field-of-view image, and the display control unit 94 changes the predetermined area image 711a by moving a virtual viewpoint on the wide-field-of-view image based on a first operation performed on the predetermined area image 711a, and scrolls the predetermined area image 711a based on a second operation performed on the predetermined area image 711a. This provides the effect of preventing the operation used for moving the virtual viewpoint from competing with the operation used for scrolling.

[0187] Here, the first operation is, for example, a drag performed by an operator (for example, viewer A) operating a mouse, or a swipe performed by the operator after touching the touch panel with one finger. The second operation is, for example, a mouse wheel operation performed by the operator, or a swipe performed by the operator after touching the touch panel with two fingers.

[0188] Note that the first operation and the second operation may be other than the above examples as long as they are different. For example, the first operation may be a swipe with two fingers and the second operation may be a swipe with one finger. Also, any number of fingers may be used as long as they are different, such as the first operation being a swipe with two fingers and the second operation being a swipe with three fingers. Also, operations may be performed using other parts of the human body instead of fingers.

[0189] (2) In addition, the present disclosure is extremely effective in the following situations:

[0190] Due to recent work style reforms and the spread of infectious diseases, there are an increasing number of cases where inspections and support work at construction sites, which were previously carried out in person, are now being carried out remotely by two-way communication of multimedia such as video (including audio) from 360-degree cameras or wearable cameras and screen sharing of documents. However, when carrying out remote inspections and support work at such sites, it has become clear that there are various challenges to carrying out work at the same or higher quality as if it were done in person.

[0191] In conventional remote communication systems, when multiple screens (or images) such as a screen displaying camera footage and an image for screen sharing are displayed simultaneously, a method in which each screen is reduced in size so that it fits on multiple screens on the display and then displayed side by side has been common, as shown in Patent Document 1. However, at construction or work sites, etc., viewer A, such as a supervisor, often views camera footage of the site side by side while viewing documents such as blueprints through screen sharing, and issues instructions to the site from a remote location. Therefore, although it would be desirable to enlarge both the screen displaying the documents and the screen displaying the camera footage, this is difficult due to display size restrictions.

[0192] In response to this, there is also a method in which the viewer A enlarges one screen by displaying it full screen on the display 507 of the communication terminal 9. However, when the viewer wants to check while comparing multiple screens, there is a problem that switching the displayed screen is cumbersome. For example, when the viewer A views a screen displaying a wide-field image obtained by shooting with a 360-degree camera while moving it to a position on the display where the viewer wants to see it, a drag (or swipe) operation for moving the virtual viewpoint in the wide-field image conflicts with a drag (or swipe) operation for scrolling to switch the displayed image to another image, making it difficult to perform the intended operation.

[0193] In contrast to this, according to the present embodiment, a first operation for changing the specific area image by moving the virtual viewpoint with respect to the wide-field image within the specific area image, and a second operation for scrolling the specific area image (a second operation different from the first operation) can be performed to prevent the respective operations from competing (conflicting), thereby enabling viewer A to easily give instructions to the site by changing the virtual viewpoint of the specific area image 711a while also referring to image 712 of the screen sharing data.

[0194] 〔supplement〕 Although the embodiments have been described above, the present invention is not limited to these embodiments, and various modifications and substitutions can be made without departing from the spirit and scope of the present invention.

[0195] For example, the configuration example in Fig. 13 and the like is divided according to main functions to facilitate understanding of the processing by the communication processing system 5, the photographing device 10, and the communication terminals 7 and 9. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the communication processing system 5, the photographing device 10, and the communication terminals 7 and 9 can be further divided into more processing units depending on the processing content. Also, one processing unit can be divided so as to include even more processes.

[0196] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0197] The terminals and the like described above are merely examples of computing environments for implementing the embodiments disclosed herein. For example, the communication processing system 5 includes a plurality of computing devices, such as a server cluster. The computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, and the like, and perform the processes disclosed herein.

[0198] Furthermore, the communication processing system 5 may be integrated into one server device, or may be divided into multiple devices.

[0199] Furthermore, the programs of the above-described embodiments can be provided domestically or internationally as a (non-transitory) recording medium such as a DVD-ROM on which the programs are stored, or as a program product.

[0200] Furthermore, the CPUs 111, 301, and 501 serving as processors may each be singular or plural.

[0201] [Additional Notes] The contents of the above-described embodiment can also be expressed by the following additional clauses.

[0202] [Additional Note 1] A display terminal that changes a predetermined area image, which is a predetermined area to be displayed in a wide-field image, A display terminal having a display control unit that changes the specified area image by moving a virtual viewpoint with respect to the wide-field image based on a first operation performed within the specified area image, and scrolls the specified area image based on a second operation performed within the specified area image.

[0203] [Additional note 2] a reception unit that receives a scroll for a display area including a plurality of images; a display control unit that scrolls the images in the display area in response to the scroll command received by the reception unit; a predetermined image among the plurality of images is a predetermined area image showing a predetermined area of ​​the wide-field-of-view image that allows moving a virtual viewpoint with respect to the wide-field-of-view image.

[0204] [Additional note 3] 3. The display terminal according to claim 2, wherein the plurality of images include the predetermined area image and an image related to screen sharing data. [Explanation of symbols]

[0205] 1. Communication Systems 3. Repeater 5. Communication Processing System 7. Communication terminals 9. Communication terminal (example of display terminal) 10. Imaging Device 51 Communication unit (an example of a transmission unit, an example of a reception unit) 52 Reception 71 Communication unit (an example of a transmission unit, an example of a reception unit, etc.) 72 Reception 74 Display control section 91 Communication unit (an example of a transmission unit, an example of a reception unit) 92 Reception 94 Display control section 507 Display (an example of a display unit) 5001 User device management DB 5002 Virtual room management database 5003 Viewpoint information management DB 7003 Viewpoint information management DB [Prior art documents] [Patent documents]

[0206] [Patent Document 1] Patent Publication No. 2022-074159

Claims

1. A display terminal having a display control unit that, based on a first operation performed within a predetermined area image, which is a predetermined area of ​​a wide-field-of-view image to be displayed, moves a virtual viewpoint with respect to the wide-field-of-view image to change the predetermined area image, and scrolls a plurality of images including the predetermined area image based on a second operation performed within the predetermined area image.

2. A display terminal according to claim 1, a display terminal having a reception unit that receives the first operation and the second operation from a same operation device.

3. The display terminal according to claim 2 , wherein the accepting unit accepts the first operation and the second operation by different operations on the operating device.

4. 4. The display terminal according to claim 3, wherein the operation device is a mouse, the first operation is a drag within the predetermined area image, and the second operation is a mouse wheel operation within the predetermined area image.

5. The reception unit determines whether the first operation or the second operation has been performed within the predetermined region image, When the reception unit determines that the first operation has been performed within the predetermined area image, the display control unit changes the predetermined area image by moving a virtual viewpoint with respect to the wide-field image, When the reception unit determines that the second operation has been performed within the predetermined area image, the display control unit scrolls the predetermined area image.

3. A display terminal according to claim 2.

6. The display terminal according to claim 1 , wherein the display control unit displays a scroll bar that accepts scrolling of the specific area image, and moves the scroll bar in response to the second operation performed within the specific area image.

7. A display terminal according to claim 1, a display terminal having a reception unit that receives the first operation and the second operation based on contact with the display terminal by an operator.

8. The display terminal according to claim 7 , wherein the accepting unit accepts the first operation and the second operation by different touch operations by the operator.

9. The display terminal of claim 8 , wherein the first operation is a first number of touches by the operator, and the second operation is a second number of touches by the operator.

10. The display terminal of claim 9 , wherein the first action is a swipe with a first number of contacts and the second action is a swipe with a second number of contacts.

11. The display terminal according to claim 10 , wherein the first number of contacts are contacts made by one finger of the operator, and the second number of contacts are contacts made by two fingers of the operator.

12. The display terminal according to claim 1, wherein the display control unit displays a reception area within the specified area image for receiving a request to execute a specified processing on the wide-field image, and when the reception area is specified, executes the specified processing in priority to the first operation and the second operation.

13. The display terminal according to claim 12 , wherein the predetermined process is enlarging or reducing the predetermined area image, or replacing the wide-field image with a predetermined image displayed in another display area.

14. A display method comprising the steps of: based on a first operation performed within a predetermined area image, which is a predetermined area of ​​a wide-field-of-view image to be displayed, a virtual viewpoint for the wide-field-of-view image is moved to change the predetermined area image; and based on a second operation performed within the predetermined area image, a display control process is executed to scroll a plurality of images including the predetermined area image.

15. A program that causes a computer to execute a display control process that, based on a first operation performed within a predetermined area image, which is a predetermined area of ​​a wide-field-of-view image to be displayed, moves a virtual viewpoint with respect to the wide-field-of-view image to change the predetermined area image, and, based on a second operation performed within the predetermined area image, scrolls a plurality of images including the predetermined area image.

16. A display terminal that displays a plurality of images in a display area during a remote conference, a reception unit that receives a scroll for the display area including the plurality of images; a display control unit that scrolls the plurality of images displayed in the display area in response to the scroll command received by the reception unit; a predetermined image among the plurality of images is a predetermined area image showing a predetermined area of ​​the wide-field-of-view image that allows moving a virtual viewpoint with respect to the wide-field-of-view image.

17. The display terminal according to claim 16 , wherein the plurality of images include the predetermined area image and an image related to screen sharing data.

18. A display terminal that displays a plurality of images in a display area during a remote conference, The display terminal includes: a reception process for receiving a scroll for the display area including the plurality of images; a display control process for scrolling the images displayed in the display area in response to the reception of the scroll request by the reception process; wherein a predetermined image among the plurality of images is a predetermined area image showing a predetermined area of ​​the wide-field-of-view image that allows a virtual viewpoint relative to the wide-field-of-view image to be moved.

19. A program for displaying a plurality of images in a display area during a remote conference, comprising: On the computer, a reception process for receiving a scroll for the display area including the plurality of images; a display control process for scrolling the images displayed in the display area in response to the reception of the scroll request by the reception process; wherein a predetermined image among the plurality of images is a predetermined area image showing a predetermined area of ​​the wide-field-of-view image in which a virtual viewpoint for the wide-field-of-view image can be moved.

Images