Imaging device, intermediary device, information processing device, transmission method, transmission / reception method, reception method, program, and information processing system

Abstract

To execute optimal data collection and data processing according to a transmission path when transmitting information from an imaging device to a destination device.SOLUTION: An imaging device 10 includes a transmission / reception unit 11 for transmitting, to a server 50, at least one of an image captured by the imaging device 10 and information other than the image via a first transmission path passing through a communication terminal 90 or a second transmission path that does not pass through the communication terminal 90, and the transmission / reception unit 11 transmits the other information determined according to the first transmission path or the second transmission path to the server 50 via the first transmission path or the second transmission path.SELECTED DRAWING: Figure 28

Description

[Technical field]

[0001] The present disclosure relates to an imaging device, an intermediary device, an information processing device, a transmission method, a transmission / reception method, a reception method, a program, and an information processing system. [Background technology]

[0002] In the case of edge devices such as cameras, there are edge devices that can use multiple upload routes to upload device information and data stored on the device (files, images, etc.) to a cloud server not only indirectly (via an intermediary device such as a smartphone or PC) but also directly (without going through an intermediary device).

[0003] Patent Document 1 describes a distributed system that provides technology for handling the internal state of a controller in an edge device by linking an edge device with a computer such as a cloud server, thereby realizing generalization of data communication for such linking and reduction in the amount of data communication.

[0004] However, although Patent Document 1 makes it possible to collect data using a plurality of routes and networks, it does not take into consideration optimal data collection and data processing in accordance with the plurality of routes. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2022-081952 A Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present invention is to execute optimal data collection and data processing in accordance with a transmission path when transmitting information from an imaging device to a destination device. [Means for solving the problem]

[0007] The photographing device of claim 1 of the present invention is connectable to a destination device and includes a transmission means for transmitting to the destination device at least one of an image photographed by the photographing device and other information other than the image via a first transmission path that passes through an intermediary device or a second transmission path that does not pass through the intermediary device, and the transmission means transmits the other information determined according to the first transmission path or the second transmission path, respectively, to the destination device via the first transmission path or the second transmission path. Effect of the Invention

[0008] According to the present invention, when transmitting information from an imaging device to a destination device, optimal data collection and data processing can be executed according to the transmission path. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing an example of an overall configuration of an image processing system according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a diagram illustrating an example of a spherical image captured by an imaging device. [Diagram 3] (A) is a hemispherical image (front) captured by the camera, (B) is a hemispherical image (back) captured by the camera, and (C) is an image represented by equirectangular projection. [Figure 4] (A) A conceptual diagram showing how a sphere is covered with an equirectangular projection image, and (B) a diagram showing a spherical image. [Diagram 5] FIG. 13 is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional sphere. [Figure 6] 13 is a diagram showing the relationship between predetermined region information and an image of a predetermined region T. FIG. [Figure 7] FIG. 2 is a diagram showing an example of a state during shooting by the shooting device. [Figure 8] FIG. 1 is a diagram illustrating an example of a spherical image. [Figure 9]FIG. 11 is a diagram illustrating an example of a planar image converted from a spherical image. [Figure 10] 1A and 1B are schematic diagrams for explaining an example of an imaging device that can be applied to the image processing system. [Figure 11] FIG. 2 is a diagram illustrating an example of a hardware configuration of an imaging apparatus according to the present embodiment. [Figure 12] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server and a communication terminal according to the present embodiment. [Figure 13] FIG. 1 is a diagram illustrating an example of a functional configuration of an image processing system according to an embodiment of the present invention. [Figure 14] FIG. 11 is a diagram illustrating another example of a functional configuration of the communication terminal according to the embodiment. [Figure 15] FIG. 4 is a conceptual diagram illustrating an example of an account management table according to the embodiment. [Figure 16] FIG. 4 is a conceptual diagram illustrating an example of a setting information management table according to the embodiment. [Figure 17] 4 is a conceptual diagram illustrating an example of a device information management table according to the embodiment. FIG. [Figure 18] FIG. 11 is a sequence diagram showing an example of a registration process for server direct transmission according to the embodiment; [Figure 19] FIG. 11 is a sequence diagram showing an example of direct transmission of an image to a server according to the embodiment. [Figure 20] FIG. 11 is a sequence diagram showing an example of direct server transmission of device information according to the embodiment; [Figure 21] 11 is a sequence diagram showing an example of a registration process for transmission and reception mediation according to the embodiment; FIG. [Figure 22] FIG. 11 is a sequence diagram showing an example of an image transmission / reception mediation process according to the embodiment. [Figure 23] 11 is a sequence diagram showing an example of a transmission / reception mediation process for device information according to the embodiment; FIG. [Figure 24] FIG. 11 is a sequence diagram showing an example of another transmission / reception mediation registration process according to the embodiment; [Diagram 25] FIG. 11 is a sequence diagram showing an example of another image transmission / reception mediation process according to the embodiment. [Figure 26] 13 is a sequence diagram showing an example of another transmission / reception mediation process of device information according to the embodiment; FIG. [Figure 27] FIG. 11 is a sequence diagram showing an example of a display process of device information according to the embodiment. [Figure 28] 10 is a flowchart showing an example of an image transmission process in the imaging device according to the present embodiment. [Figure 29] 5 is a flowchart showing an example of an image transmission / reception process in the communication terminal and an image reception process in the server according to the embodiment. [Diagram 30] 10 is a flowchart illustrating an example of a device information transmission process in the imaging device according to the present embodiment. [Diagram 31] FIG. 4 is a diagram showing an example of a display screen of device information according to the embodiment; [Diagram 32] FIG. 4 is a diagram showing an example of a display screen of device information according to the embodiment; [Diagram 33] FIG. 13 is a diagram showing another example of the display screen of device information according to the embodiment. [Diagram 34] FIG. 11 is a diagram showing an example of an overall configuration of an image processing system according to a modified example of an embodiment of the present invention. [Diagram 35] FIG. 11 is a diagram showing an example of an overall configuration of an image processing system according to a second modified example of the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated description will be omitted.

[0011] ●Embodiment● ●Outline of image processing system Fig. 1 is a diagram showing an example of the overall configuration of an image processing system according to an embodiment of the present invention. The image processing system 1 shown in Fig. 1 is an example of an information processing system, and is a system that performs image processing on captured images to allow viewers to view the internal space of structures such as real estate and architectural properties, and structures such as construction and civil engineering structures online.

[0012] As shown in FIG. 1, the image processing system 1 includes a photographing device 10, a server 50, and a communication terminal 90. The server 50 and the communication terminal 90 constituting the image processing system 1 can communicate with each other via a communication network 100. The photographing device 10 can communicate with the communication terminal 90 by short-distance wireless communication such as Wi-Fi, but may also communicate with the server 50 and the communication terminal 90 via the communication network 100. The communication network 100 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), and the like. Note that the communication network 100 may include not only wired communication but also networks using wireless communication such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0013] The server 50 is an example of an information processing device and a destination device, and is a server computer that performs image processing on captured images of the internal space of a structure such as real estate or a building, which is a specific base, or a structure such as a construction or civil engineering structure. The server 50, for example, acquires captured images captured by the image capture device 10, and generates tour images for providing a user with a virtual tour using the acquired captured images. Here, the virtual tour is a content that allows a user to view a real estate property or the like as if they were actually viewing the property at the site. In addition, the tour image is an image for viewing that is generated using a plurality of captured images captured by the image capture device 10, and allows the user to virtually move around the base shown in the captured image by a user operation. Such a virtual tour is not limited to the internal space of a structure of a real estate property or a construction site, but can also be suitably realized in the external space of a structure such as a tourist spot or a theme park. In other words, the internal space of a structure can also be regarded as being within a specific area. When a virtual tour is realized in the external space of a structure, the present embodiment can be realized by replacing a map showing the internal space of a structure such as real estate or a building, which will be described later, with, for example, a tourist spot map introducing a tourist spot or an area map of a theme park.

[0014] The server 50 may be configured by one server computer or may be configured by multiple server computers. Although the server 50 will be described as a server computer that exists in a cloud environment, the server 50 may be a server that exists in an on-premise environment.

[0015] The image capturing device 10 is an example of an information processing device or a transmission source device, and is a special digital camera (omnidirectional image capturing device) capable of capturing an image of a celestial sphere (360° in both the circumferential and vertical directions) by capturing an image of a space in which a structure such as a real estate property exists in all directions at an image capturing base. In the following, an example of a structure such as a real estate property will be mainly described, but as described above, the image capturing device 10 may be an internal space of a structure such as an architectural property, or a structure such as a construction or civil engineering structure.

[0016] Alternatively, instead of the photographing device 10, for example, a communication terminal 90 having a function capable of acquiring a celestial sphere image may acquire the celestial sphere image, or a camera attachment capable of acquiring a celestial sphere image may be connected to the communication terminal 90 and the celestial sphere image may be acquired by the communication terminal 90 and the camera attachment. In other words, the transmission source device may be any device that transmits the acquired celestial sphere image to the server 50.

[0017] Note that a celestial sphere image refers to an image with a so-called solid angle of 4πsr [sr: steradian]. In this specification, an image in which a part of a celestial sphere image is missing is also called a celestial sphere image for convenience. For example, an image in which a part of a direction directly above or directly below the celestial sphere imaging device, a part of a vertically upward or downward direction of the celestial sphere image, or a part of a predetermined region of the celestial sphere image is missing.

[0018] For example, there may be usage scenarios where the area directly above or below a subject in a spherical image is not looked at very carefully when viewing the spherical image. In such cases, it is conceivable to design the image sensor and optical system so as not to capture that area in the first place, not to display an image, or to display a logo or the like in that area, or to not display the spherical image itself.

[0019] The photographing device 10 is used, for example, by a real estate agent who manages or sells real estate properties. The photographing device 10 may be a wide-angle camera or a stereo camera capable of acquiring a wide-angle image having a viewing angle of a predetermined value or more. A wide-angle image is generally an image captured using a wide-angle lens (wide-angle image), and is an image captured using a lens that can capture a wider range than the human eye can sense. In other words, the photographing device 10 is a photographing means capable of acquiring an image captured using a lens with a focal length shorter than a predetermined value (a celestial sphere image, a wide-angle image). A wide-angle image generally means an image captured using a lens with a focal length of 35 mm or less in 35 mm film equivalent. The photographing device 10 may have a panoramic photographing function as a photographing function, and may be one that captures panoramic images.

[0020] Here, when transmitting an image from the photographing device 10 to the server 50, there are cases where the image cannot be transmitted directly from the photographing device 10 to the server 50 due to the communication environment, etc. In such cases, the image is transmitted from the photographing device 10 to the server 50 via the communication terminal 90.

[0021] The communication terminal 90 is an example of an intermediary device or an information processing device, and is a computer such as a tablet terminal. The communication terminal 90 is used, for example, by the same real estate agent as the photographing device 10. The communication terminal 90 is installed with, for example, a dedicated application for issuing photographing instructions to the photographing device 10 and viewing images provided by the server 50, and an application related to transmission and reception that mediates the transmission and reception of information between the photographing device 10 and the server 50. The communication terminal 90 may be configured to issue photographing instructions and view images by accessing a dedicated website using a web browser, without using a dedicated application. The photographing instructions and image viewing may be performed by a different communication terminal 90. Hereinafter, the application will be simply referred to as an app.

[0022] In addition, the communication terminal 90 is not limited to a tablet terminal, but may be, for example, a PC, a smartphone, a wearable terminal, an HMD (head mounted display), or an IWB (Interactive White Board: an electronic whiteboard with a blackboard function that allows mutual communication), etc.

[0023] In the image processing system 1 described above, the photographing device 10 and the server 50 are equipped with APIs, and the communication terminal 90 is equipped with an application that mediates the transmission and reception of information between the photographing device 10 and the server 50. This makes it possible to transmit images from the photographing device 10 to the server 50 via the communication terminal 90. However, there may be restrictions on transmitting device information of the photographing device 10, such as remaining battery level and remaining storage capacity, to the server 50.

[0024] For example, if the type, data format, communication protocol, etc. of device information transmitted from the photographing device 10 to the server 50 is changed, due to a time lag between the development of a functional update of the photographing device 10 and a functional update of the application of the communication terminal 90, the types of device information transmitted from the photographing device 10 to the server 50 via the communication terminal 90 may be fewer than the types of device information transmitted directly from the photographing device 10 to the server 50.

[0025] This problem is particularly pronounced when using an app developed by another company, and there is a possibility that the types of device information transmitted from the photographing device 10 to the server 50 via a communication terminal 90 equipped with an app developed by another company will be fewer than the types of device information transmitted from the photographing device 10 to the server 50 via a communication terminal 90 equipped with an app developed by the company itself.

[0026] In the above situation, if the image capturing device 10 transmits a wide variety of device information uniformly regardless of the type of transmission path when transmitting directly to the server 50, a reception error will occur when the information is transmitted via the communication terminal 90.

[0027] On the other hand, if the photographing device 10 transmits a small number of types of device information uniformly regardless of the type of transmission path in order to transmit via the communication terminal 90 equipped with an app developed by another company, it will be restricted from receiving device information that would normally be receivable when transmitting directly to the server 50.

[0028] The above problem is not limited to the device information of the photographing device 10, but is also the same for the image identification information. That is, the image identification information indicates whether or not the image transmitted by the photographing device 10 has been subjected to image processing, or the type of image processing that has been performed, but depending on the compatibility of the app, there is a possibility that the image identification information cannot be transmitted or received via the communication terminal 90.

[0029] In this situation, if the photographing device 10 transmits image identification information uniformly regardless of the type of transmission path in accordance with the direct transmission to the server 50, a reception error will occur if the information is transmitted via the communication terminal 90.

[0030] On the other hand, if the photographing device 10 does not transmit image identification information uniformly regardless of the type of transmission path in accordance with the communication terminal 90 equipped with the latest application developed by another company, the server 50 cannot determine whether an image is processed or not and what type of image is processed, so there is a restriction that no image processing is performed on the photographing device 10 side, and all image processing must be performed on the server 50 side. The above problem is not limited to cases where the photographing device 10 and the server 50 are equipped with APIs, but is similar when the communication terminal 90 is equipped with an API.

[0031] The present embodiment aims to perform optimal data collection and data processing depending on the transmission path when transmitting device information, image identification information, and other information from the photographing device to the destination device as described above.

[0032] ○Outline of the imaging device○ Here, an overview of the photographing device 10 constituting the image processing system 1 will be described with reference to Figs. 2 to 11. Fig. 2 is a diagram showing an example of a celestial sphere image photographed by the photographing device. The image shown in Fig. 2 is a celestial sphere image photographed by the photographing device 10 of a room of a real estate property, which is an example of the internal space of a structure. A celestial sphere image is suitable for viewing real estate properties and the like, since the interior of a room can be photographed in all directions. There are various forms of celestial sphere images, but they are often generated by the equirectangular projection method described later. The advantages of an image generated by this equirectangular projection method are that the outer shape of the image is rectangular, making it efficient and easy to store image data, and that there is little distortion near the equator and vertical straight lines are not distorted, making it look relatively natural.

[0033] ○How to generate spherical images Next, a method for generating a celestial sphere image will be described with reference to Fig. 3 to Fig. 9. First, an outline of a process for generating a celestial sphere image from an image captured by the image capture device 10 will be described with reference to Fig. 3 and Fig. 4. Fig. 3(A) is a diagram showing a hemispherical image (front side) captured by the image capture device, Fig. 3(B) is a hemispherical image (rear side) captured by the image capture device, and Fig. 3(C) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). Fig. 4(A) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 4(B) is a diagram showing a celestial sphere image.

[0034] The photographing device 10 is provided with imaging elements on both the front side (front side) and the back side (rear side). These imaging elements (image sensors) are used in conjunction with optical components such as lenses capable of photographing hemispherical images (angle of view of 180° or more). The photographing device 10 can obtain two hemispherical images by capturing images of subjects around the user using the two imaging elements, respectively.

[0035] 3(A) and (B), the image captured by the imaging element of the photographing device 10 is a curved hemispherical image (front and rear). The photographing device 10 then synthesizes the hemispherical image (front) and the hemispherical image (rear) flipped 180 degrees to create an equirectangular projection image EC as shown in FIG. 3(C).

[0036] Then, the photographing device 10 uses OpenGL ES (Open Graphics Library for Embedded Systems) to paste an equirectangular projection image EC so as to cover the sphere as shown in FIG. 4A, and creates a celestial sphere image (celestial sphere panoramic image) CE as shown in FIG. 4B. In this way, the celestial sphere image CE is expressed as an image in which the equirectangular projection image EC faces the center of the sphere. Note that OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. In addition, the celestial sphere image CE may be a still image or a video. Furthermore, the conversion method is not limited to OpenGL ES, and may be any method capable of converting a hemispherical image into an equirectangular projection, for example, a calculation by a CPU or a calculation by OpenCL.

[0037] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to people when they see it. Therefore, the imaging device 10 can display a predetermined region T (hereinafter, referred to as a "predetermined region image") that is a part of the spherical image CE as a planar image with little curvature, thereby enabling a display that does not give a sense of incongruity to people. This will be described with reference to Figs. 5 and 6.

[0038] FIG. 5 is a diagram showing the positions of the virtual camera and the predetermined area when the celestial sphere image is a three-dimensional 3D sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the celestial sphere image CE displayed as a three-dimensional 3D sphere. FIG. 5 shows the celestial sphere image CE as a three-dimensional 3D sphere CS. If the celestial sphere image CE generated in this way is a 3D sphere CS, as shown in FIG. 5, the virtual camera IC is located inside the celestial sphere image CE. The predetermined area T in the celestial sphere image CE is the shooting area of ​​the virtual camera IC, and is specified by the predetermined area information indicating the shooting direction and the angle of view of the virtual camera IC in the three-dimensional virtual space including the celestial sphere image CE. In addition, the zoom of the predetermined area T can also be expressed by moving the virtual camera IC closer to or farther away from the celestial sphere image CE. The predetermined area image Q is an image of the predetermined area T in the celestial sphere image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the celestial sphere image CE.

[0039] The predetermined area image Q is then displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC. The following description will be given using the shooting direction (ea, aa) and angle of view (α) of the virtual camera IC. Note that the predetermined area T may be indicated by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the angle of view α and the distance f.

[0040] Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 6. FIG. 6 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 6, "ea" indicates an elevation angle, "aa" indicates an azimuth angle, and "α" indicates an angle of view. That is, the attitude of the virtual camera IC is changed so that the gaze point of the virtual camera IC indicated by the shooting direction (ea, aa) becomes the center point CP(x, y) of the predetermined area T, which is the shooting area of ​​the virtual camera IC. As shown in FIG. 6, the center point CP(x, y) when the diagonal angle of view of the predetermined area T represented by the angle of view α of the virtual camera IC is α becomes the parameter ((x, y)) of the predetermined area information. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. f is the distance from the virtual camera IC to the center point CP(x, y). L is the distance between any vertex of the predetermined area T and the center point CP(x, y) (2L is the diagonal line). In FIG. 6, the trigonometric function shown in the following formula 1 generally holds.

[0041]

number

[0042] Next, a state of photographing by the photographing device 10 will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of a state of photographing by the photographing device. In order to photograph a room of a real estate property or the like so that the entire room can be viewed, it is preferable to install the photographing device 10 at a position close to the height of human eyes. For this reason, as shown in FIG. 7, the photographing device 10 is generally fixed to a support member 20 such as a monopod or a tripod for photographing. As described above, the photographing device 10 is a celestial sphere photographing device capable of acquiring light rays in all directions around the entire periphery, and it can also be said that the photographing device 10 acquires an image (a celestial sphere image CE) on a unit sphere around the photographing device 10. When the photographing direction of the photographing device 10 is determined, the coordinates of the celestial sphere image are determined. For example, in FIG. 7, point A is at a distance (d, -h) from the center point C of the photographing device 10. If the angle between the line segment AC and the horizontal direction is θ, the angle θ can be expressed by the following (Equation 2).

[0043]

number

[0044] Assuming that point A is at a depression angle θ, the distance d between points A and B can be expressed by the following (Equation 3) using the installation height h of the image capturing device 10.

[0045]

number

[0046] Here, a process of converting position information on a celestial sphere image into coordinates on a planar image converted from the celestial sphere image will be outlined. Fig. 8 is a diagram for explaining an example of a celestial sphere image. Fig. 8(A) is a diagram showing the hemispherical image shown in Fig. 3(A) by connecting points where the horizontal and vertical incidence angles with respect to the optical axis are equal with lines. Hereinafter, the horizontal incidence angle with respect to the optical axis is referred to as "θ", and the vertical incidence angle with respect to the optical axis is referred to as "φ".

[0047] Also, Fig. 9(A) is a diagram for explaining an example of an image processed by equirectangular projection. Specifically, the image shown in Fig. 8 is made to correspond to a pre-generated LUT (Look Up Table) or the like, processed by equirectangular projection, and the processed images shown in Fig. 8(A) and Fig. 8(B) are synthesized, whereby the planar image shown in Fig. 9(A) corresponding to the omnidirectional image is generated by the photographing device 10. The equirectangular projection image EC shown in Fig. 3(C) is an example of the planar image shown in Fig. 9(A).

[0048] As shown in FIG. 9(A), in an image processed by equirectangular projection, latitude (θ) and longitude (φ) are orthogonal to each other. In the example shown in FIG. 9(A), the center of the image is (0,0), and the latitude direction is expressed as -90 to +90, and the longitude direction is expressed as -180 to +180, so that any position in the omnidirectional image can be indicated. For example, the coordinates of the upper left corner of the image are (-180, -90). The coordinates of the omnidirectional image may be expressed in a format using 360 degrees as shown in FIG. 9(A), or may be expressed in radians or in the number of pixels as in a real image. The coordinates of the omnidirectional image may be converted and expressed as two-dimensional coordinates (x, y) as shown in FIG. 9(B).

[0049] Note that the synthesis process for the planar image shown in Fig. 9(A) or Fig. 9(B) is not limited to the process of simply arranging the hemispherical images shown in Fig. 8(A) and Fig. 8(B) consecutively. For example, if the horizontal center of the omnidirectional image is not θ=180°, in the synthesis process, the image capturing device 10 first preprocesses the hemispherical image shown in Fig. 3(C) and arranges it at the center of the omnidirectional image. Next, the image capturing device 10 may divide the hemispherical image shown in Fig. 3(B) into left and right parts of the image to be generated into a size that allows the preprocessed image to be arranged on the left and right parts, and synthesize the hemispherical images to generate the equirectangular projection image EC shown in Fig. 3(C).

[0050] Also, in the planar image shown in Fig. 9(A), the point corresponding to the pole (PL1 or PL2) of the hemispherical image (spherical image) shown in Fig. 8(A) and Fig. 8(B) is the line segment CT1 or CT2. This is because, as shown in Fig. 4(A) and Fig. 4(B), the celestial sphere image (for example, the celestial sphere image CE) is created by pasting the planar image (equirectangular projection image EC) shown in Fig. 9(A) onto a spherical surface by using OpenGL ES.

[0051] Examples of imaging devices that can be used with image processing systems FIG. 10 is a schematic diagram for explaining an example of an imaging device applicable to an image processing system. FIG. 10(A) shows a special imaging device equipped with a plurality of imaging elements capable of generating an omnidirectional image by the above-mentioned generation method. The special imaging device uses an optical system composed of a wide-angle lens with a wide angle of view, a fisheye lens, etc., and can acquire an image in which all directions are captured by combining the outputs of the plurality of imaging elements. FIG. 10(B) shows a general imaging device, which is a so-called normal camera. The general imaging device is, for example, a normal digital camera or a mobile terminal such as a smartphone equipped with a camera. The photographer takes pictures while holding the general imaging device in his / her hand and rotating it. The general imaging device can acquire an omnidirectional image by synthesizing the obtained images. Both the special imaging device and the general imaging device generate a final captured image by stitching together multiple captured images by image processing (stitching process). It is preferable that the optical center of the imaging device 10 for obtaining multiple captured images is the same.

[0052] ●Hardware configuration Next, the hardware configuration of each device or terminal constituting the image processing system according to the embodiment will be described with reference to Figures 11 and 12. Note that components may be added or deleted from the hardware configurations shown in Figures 11 and 12 as necessary.

[0053] ○Hardware configuration of the imaging device○ First, the hardware configuration of the imaging device 10 will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of the hardware configuration of the imaging device. In the following, the imaging device 10 is an omnidirectional (all-directional) imaging device using two imaging elements, but the number of imaging elements may be two or more. In addition, the imaging device does not necessarily have to be a device dedicated to omnidirectional imaging, and an omnidirectional imaging unit may be attached to a normal digital camera, smartphone, or the like to have substantially the same functions as the imaging device 10.

[0054] 11, the photographing device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F (Interface) 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121. The photographing device 10 also includes an ISP (Image Signal Processor) that is a processor that processes video signals.

[0055] Among these, the imaging unit 101 includes optical systems (wide-angle lenses or so-called fisheye lenses) 102a and 102b (hereinafter referred to as lenses 102 when there is no need to distinguish between them) each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to each lens. Here, a combination of one optical system and one imaging element is called an imaging optical system, and the imaging device 10 can be configured by arranging two imaging optical systems facing each other. It is also possible to configure the imaging device 10 by adopting two or more imaging optical systems. The imaging elements 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image by the lenses 102a and 102b into image data of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands or parameters required for the operation of the imaging elements are set.

[0056] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. In addition, the ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the input / output I / F 116, the short-range communication circuit 117, the electronic compass 118, the gyro sensor 119, the acceleration sensor 120, and the network I / F 121 are also connected to the bus 110.

[0057] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs a predetermined processing on each image data, and then synthesizes the image data to create equirectangular projection image data such as that shown in Figure 3(C).

[0058] The imaging control unit 105 generally sets commands and the like in the register groups of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

[0059] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function corresponding to video display on a display (for example, a display of an external terminal such as a smartphone that performs short-distance communication with the imaging device 10 using the short-distance communication circuit 117). In this case, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames / second).

[0060] As described later, the imaging control unit 105 also functions as a synchronization control means for synchronizing the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through an I / F bus and performs a predetermined process on the sound data.

[0061] The CPU 111 controls the overall operation of the photographing device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104, and data of a processed equirectangular projection image.

[0062] The operation unit 115 is a collective term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, etc. A user operates the operation unit 115 to input various shooting modes, shooting conditions, etc.

[0063] The input / output I / F 116 is a general term for an interface circuit (such as an SD memory card I / F or USB I / F) with an external medium such as an SD memory card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as necessary.

[0064] The short-range communication circuit 117 communicates with an external terminal (device) by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi via an antenna 117a provided in the image capturing device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0065] The electronic compass 118 calculates the direction of the image capturing device 10 from the earth's magnetism and outputs the direction information. This direction information is an example of related information (metadata) according to Exif and is used for image processing such as image correction of the captured image. The related information also includes each data of the image capture date and time and the data capacity of the image data. The gyro sensor 119 is a sensor that detects angle changes (roll angle, pitch angle, yaw angle) accompanying the movement of the image capturing device 10. The angle changes are an example of related information (metadata) according to Exif and are used for image processing such as image correction of the captured image. The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The image capturing device 10 calculates the attitude (angle with respect to the direction of gravity) of the own device (image capturing device 10) based on the acceleration detected by the acceleration sensor 120. The accuracy of image correction is improved by providing the acceleration sensor 120 in the image capturing device 10. The network I / F 121 is an interface for performing data communication using a communication network 100 such as the Internet via a router or the like.

[0066] ○Server hardware configuration○ First, the hardware configuration of the server 50 will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of the hardware configuration of the server. Each hardware component of the server 50 is indicated by a reference number in the 500 series. The server 50 is constructed by a computer, and includes a CPU 501, a ROM 502, a RAM (Random Access Memory) 503, a HD (Hard Disk) 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516, as shown in Fig. 12.

[0067] Among these, the CPU 501 controls the operation of the entire server 50. The ROM 502 stores programs used to drive the CPU 501, such as an IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data from the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, a menu, a window, a character, or an image. The display 506 may be a touch panel display equipped with an input means. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB memory. The network I / F 509 is an interface for data communication using the communication network 100. The bus line 510 is an address bus or a data bus for electrically connecting each component such as the CPU 501 shown in FIG. 12.

[0068] The keyboard 511 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting or executing various instructions, selecting a processing target, moving a cursor, etc. The input means may be not only the keyboard 511 and the pointing device 512, but also a touch panel or a voice input device, etc. The DVD-RW drive 514 controls reading or writing of various data from a DVD-RW 513 as an example of a removable recording medium. The removable recording medium is not limited to a DVD-RW, and may be a DVD-R or a Blu-ray (registered trademark) Disc, etc. The media I / F 516 controls reading or writing (storing) of data from a recording medium 515 such as a flash memory.

[0069] ○Hardware configuration of communication terminal○ Fig. 12 also shows an example of the hardware configuration of the communication terminal 90. Each hardware configuration of the communication terminal 90 is indicated by a reference number in the 900 series in parentheses. The communication terminal 90 is constructed by a computer, and as shown in Fig. 12, has the same configuration as the server 50, so a description of each hardware configuration will be omitted. In addition to the same configuration as the server 50, the communication terminal 90 has a short-range communication circuit 917 and an antenna 917a of the short-range communication circuit 917. The short-range communication circuit 917 is a communication circuit such as NFC, Bluetooth, or Wi-Fi.

[0070] Each of the above programs may be recorded in a computer-readable recording medium as an installable or executable file and distributed. Examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray Disc, an SD card, and a USB memory. The recording medium may be provided domestically or internationally as a program product. For example, the server 50 realizes the image processing method according to the present invention by executing the program according to the present invention.

[0071] ●Function configuration Next, the functional configuration of the image processing system according to the embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram showing an example of the functional configuration of the image processing system. Fig. 13 shows devices or terminals shown in Fig. 1 that are related to the processes or operations described below.

[0072] ○Functional configuration of the imaging device○ First, the functional configuration of the photographing device 10 will be described with reference to FIG. 13. The photographing device 10 has a transmitting / receiving unit 11, an operation receiving unit 12, a photographing control unit 13, a moving image photographing unit 14, a still image photographing unit 15, an image processing unit 16, a data management unit 17, and a storage / reading unit 19. Each of these units is a function or means realized by operating any of the components shown in FIG. 11 by an instruction from the CPU 111 according to a program for the photographing device expanded from the SRAM 113 onto the DRAM 114. The photographing device 10 also has a storage unit 1000 constructed by the ROM 112, the SRAM 113, and the DRAM 114 shown in FIG. 11. The storage unit 1000 stores a GUID (Globally Unique Identifier) ​​of the own device.

[0073] The transmitting / receiving unit 11 is an example of a transmitting means and a receiving means, and is mainly realized by the processing of the CPU 111, and communicates various data or information with other devices or terminals. Furthermore, the transmitting / receiving unit 11 uses the network I / F 121 to communicate data with other devices or terminals via the communication network 100.

[0074] The operation reception unit 12 is mainly realized by the processing of the CPU 111 on the operation unit 115, and receives various selections or inputs from the user who is the photographer.

[0075] The shooting control unit 13 is mainly realized by the processing of the CPU 111 for the imaging unit 101, the image processing unit 104, and the imaging control unit 105, and captures an image of a subject such as a landscape, and acquires captured image data. The shooting control unit 13 performs shooting by, for example, switching between video shooting by the video shooting unit 14 and still image shooting by the still image shooting unit 15 in a time-division manner.

[0076] The video shooting unit 14 is mainly realized by the processing of the CPU 111 on the imaging unit 101, the image processing unit 104, and the imaging control unit 105, and performs video shooting by the imaging device 10. The video shooting unit 14 shoots moving images while moving within a structure such as a real estate property that is a predetermined base or a space in which the structure is arranged. The video shooting unit 14 shoots video in continuous frames at low resolution while the photographer holding the imaging device 10 is moving, and stores the captured image data in the storage unit 1000. The video shooting unit 14 shoots video while the photographer holding the imaging device 10 is moving from a first point to a second point within a real estate property that is a predetermined base, for example.

[0077] The still image shooting section 15 is mainly realized by the processing of the imaging unit 101, and captures an image of a subject such as a landscape, and performs still image shooting by the imaging device 10. The still image shooting section 15 captures a plurality of still images captured at different shooting positions within a structure such as a real estate property, which is a predetermined base, or a space in which the structure is arranged. The still image shooting section 15 captures still images (photos) with a higher resolution than the moving images captured by the video shooting section 14, for example, and stores the captured image data in the storage section 1000. The still image captured by the still image shooting section 15 may be a single frame image, or may be an HDR (High Dynamic Range) image obtained by synthesizing a plurality of images. Video shooting is performed during movement within a predetermined base, and still image shooting is performed during stationary time within the predetermined base.

[0078] Here, the shooting specifications required for video shooting and still image shooting are different according to the shooting purpose. In the case of video shooting, continuous images (continuous frames) at a high frame rate are acquired for use in position estimation, while high resolution and color layers of the captured images are not essential, and low resolution and grayscale are sufficient. In addition, in the case of still image shooting, while acquisition of continuous frames is not necessary, shooting with high resolution, color information (RGB), and high dynamic range is required. For the purpose of acquiring images for viewing, it is preferable that the still images captured by the still image shooting unit 15 have a high resolution of, for example, 4K resolution or higher. On the other hand, the video images captured by the video shooting unit 14 are images used for position estimation, and only the subject appearing in the video images needs to be identifiable, so that the resolution is lower than that of the still images. The video images may have a resolution of, for example, about 480p or lower. The shooting device 10 can reduce the overall data amount in tour shooting by shooting low-resolution video images.

[0079] The image processing unit 16 is mainly realized by the processing of the CPU 111, and performs various image processing on still images captured by the still image capturing unit 15 and moving images captured by the moving image capturing unit .

[0080] The data management unit 17 is an example of a determination means, and is mainly realized by the processing of the CPU 111, and performs processing for managing various types of device information.

[0081] The storage / reading unit 19 is an example of a setting means, and is mainly realized by the processing of the CPU 111, and stores various data (or information) in the storage unit 1000 and reads various data (or information) from the storage unit 1000. The storage unit 1000 also stores image data captured by the video capture unit 14 and the still image capture unit 15. The image data stored in the storage unit 1000 is associated with the capture time of the captured image as metadata.

[0082] A setting information management DB 1001, an image management DB 1002, and a device information management DB 1003 are constructed in the storage unit 1000. The setting information management DB 1001 is composed of a setting information management table, which will be described later, and stores and manages setting information and the like for image processing executed by the image processing unit 16 on still images and moving images.

[0083] The image management DB 1002 stores and manages the captured images captured by the imaging device 10 and the processed images obtained by performing image processing on the captured images by the image processing unit 16, in association with the image IDs. The captured images and the processed images may be stored in a storage medium such as an SD memory card, which is the above-mentioned external medium.

[0084] The device information management DB 1003 stores and manages various types of device information.

[0085] ○Server Functional Configuration○ First, the functional configuration of the server 50 will be described with reference to Fig. 13. The server 50 has a transmitting / receiving unit 51, a receiving unit 52, an authentication unit 53, a data processing unit 54, a determination unit 55, an image processing unit 57, and a storage / reading unit 59. Each of these units is a function or means realized by operating any of the components shown in Fig. 12 by an instruction from the CPU 501 in accordance with the server program loaded from the HD 504 onto the RAM 503. The server 50 also has a storage unit 5000 constructed by the ROM 502, RAM 503, and HD 504 shown in Fig. 12.

[0086] The transmitting / receiving unit 51 is an example of a transmitting means and a receiving means, and is mainly realized by processing of the CPU 501 on the network I / F 509, and transmits and receives various data or information to and from other devices or terminals via the communication network 100. The transmitting / receiving unit 51 receives (acquires), for example, a moving image captured by the imaging device 10 from the imaging device 10 or the communication terminal 90. The transmitting / receiving unit 51 also receives (acquires), for example, a still image captured by the imaging device 10 from the imaging device 10 or the communication terminal 90.

[0087] The reception unit 52 is realized mainly by the processing of the CPU 501 on the keyboard 511 or the pointing device 512, and receives various selections or inputs from the user.

[0088] The authentication unit 53 is realized mainly by the processing of the CPU 501, and performs various authentications.

[0089] The data processing unit 54 is realized mainly by the processing of the CPU 501, and performs data processing of various types of device information.

[0090] The determination unit 55 is realized by the processing of the CPU 501, and performs various determination processes.

[0091] The image processing unit 57 is realized by the processing of the CPU 501, and performs various image processing on still images and moving images.

[0092] The storage / reading unit 59 is an example of a setting means, and is realized mainly by processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000 .

[0093] The storage unit 5000 stores a setting information management DB 5001, an image management DB 5002, a device information management DB 5003, and an account management DB 5004. The setting information management DB 5001 is made up of a setting information management table, which will be described later, and stores and manages setting information and the like for image processing executed by the image processing unit 57 for still images and moving images.

[0094] The image management DB 5002 stores and manages processed images that have been processed by the photographing device 10 or the communication terminal 90 and received from the photographing device 10 or the communication terminal 90, and processed images that have been subjected to image processing by the image processing unit 57 on the received processed images, in association with their respective image IDs.

[0095] The processed image that has been subjected to image processing by the image processing unit 57 may be transmitted to the photographing device 10 or the communication terminal 90 and stored in the image management DB 1002 or 9002 on the photographing device 10 or the communication terminal 90 side, or in a storage medium such as an SD memory card.

[0096] The device information management DB 5003 stores and manages various types of device information. The device information is an example of information other than images. The account management DB 5004 stores and manages account information such as user IDs and management information for managing the image capture device 10.

[0097] ○Functional configuration of communication terminal○ Next, the functional configuration of the communication terminal 90 will be described with reference to Fig. 13. The communication terminal 90 has a transmission / reception unit 91, a reception unit 92, a display control unit 93, an image processing unit 94, a data control unit 95, an application function unit 96, an authentication unit 97, and a storage / readout unit 99. Each of these units is a function or means realized by operating any of the components shown in Fig. 13 by an instruction from the CPU 901 according to a program for the communication terminal loaded from the HD 904 onto the RAM 903. The communication terminal 90 also has a storage unit 9000 constructed by the ROM 902, RAM 903, and HD 904 shown in Fig. 13. The storage unit 9000 stores various applications.

[0098] The transmission / reception unit 91 is an example of a transmitting means and a receiving means, and is mainly realized by processing of the CPU 901 on the network I / F 909, and transmits and receives various data or information with other devices or terminals via the communication network 100.

[0099] The reception unit 92 is an example of a setting unit, and is realized mainly by processing of the CPU 901 via the keyboard 911 or the pointing device 912, and receives various selections or inputs from the user.

[0100] The display control unit 93 is mainly realized by the processing of the CPU 901, and causes various images, characters, etc. to be displayed on the display 906. The display control unit 93 accesses the server 50 using, for example, a Web browser or a dedicated application, and causes an image corresponding to data distributed from the server 50 to be displayed on the display 906.

[0101] The image processing unit 94 is realized mainly by the processing of the CPU 901, and performs various types of image processing on still images and moving images.

[0102] The data control unit 95 is realized mainly by the processing of the CPU 901, and controls various types of data.

[0103] The application function unit 96 is realized mainly by the processing of the CPU 901 , and executes the functions of various applications stored in the storage unit 9000 .

[0104] Specifically, the application function unit 96 executes the functions of an application related to transmission and reception that mediates the transmission and reception of information between the photographing device 10 and the server 50, and cooperates with the transmission and reception unit 91, the data control unit 95, the transmission and reception unit 11 of the photographing device 10, the data management unit 17, the transmission and reception unit 51 and data processing unit 54 of the server 50, etc. to control the data and information that can be communicated between the photographing device 10, the communication terminal 90, and the server 50.

[0105] The authentication unit 97 is realized mainly by the processing of the CPU 901, and performs various authentications.

[0106] The storage / reading unit 99 is realized mainly by the processing of the CPU 901 , and stores various data (or information) in the storage unit 9000 and reads various data (or information) from the storage unit 9000 .

[0107] A setting information management DB 9001 and an image management DB 9002 are constructed in the storage unit 9000. The setting information management DB 9001 is composed of a setting information management table, which will be described later, and stores and manages setting information and the like for image processing executed by the image processing unit 94 on still images and moving images.

[0108] The image management DB 9002 stores and manages the captured images captured by the photographing device 10 and received from the photographing device 10, and the processed images obtained by performing image processing on the captured images by the image processing unit 94, in association with the image IDs. The captured images and the processed images may be stored in a storage medium such as an SD memory card, which is an external medium.

[0109] FIG. 14 is a diagram illustrating another example of a functional configuration of the communication terminal.

[0110] 13, a communication terminal 90A shown in Fig. 14(a) has a transmission / reception unit 91, a reception unit 92, a display control unit 93, a data control unit 95, an application function unit 96, and a storage / readout unit 99, but does not have an image processing unit 94 or an authentication unit 97. Also, an image management DB 9002 is configured in a storage unit 9000, but a setting information management DB 9001 is not configured.

[0111] 13, a communication terminal 90B shown in Fig. 14(b) has a transmission / reception unit 91, a reception unit 92, a data control unit 95, an application function unit 96, and a storage / readout unit 99, but does not have a display control unit 93, an image processing unit 94, or an authentication unit 97. Also, an image management DB 9002 is configured in a storage unit 9000, but a setting information management DB 9001 is not configured.

[0112] The communication terminals 90A and 90B are examples of communication terminals equipped with applications relating to transmission and reception developed by other companies.

[0113] FIG. 15 is a conceptual diagram showing an example of an account management table according to the present embodiment.

[0114] FIG. 15 shows an account management table constituting the account management DB 5004 constructed in the storage unit 5000 of the server 50 shown in FIG.

[0115] In the account management table constituting the account management DB 5004 shown in FIG. 15(a), a team ID for identifying a team using the application, a user ID for identifying a user belonging to the team, a user's email address, and a password are managed in association with the application ID.

[0116] The application ID is information for identifying the type of application related to transmission and reception stored in the storage unit 9000 of the communication terminal 90.

[0117] Here, the application function unit 96 of the communication terminal 90 executes the function of an application related to transmission and reception that mediates the transmission and reception of information between the photographing device 10 and the server 50, and controls the data and information that can be communicated between the photographing device 10, the communication terminal 90, and the server 90. Therefore, it can be said that the data and information that can be communicated are determined according to the type of application related to transmission and reception.

[0118] The application ID "RICOH-APP" is an example of an application developed in-house, and as will be described later, there are many types of information that can be transmitted and received between the image capturing device 10 and the server 90 related to the company's system.

[0119] In the account management table constituting the account management DB 5004 shown in Figure 15 (b), a virtual device ID for identifying the photographing device 10, a serial number of the photographing device 10, image data (path to the image management DB 5002), and device information (path to the device information management DB 5003) are managed in association with an application ID, a team ID, and a user ID.

[0120] In the account management table constituting the account management DB 5004 shown in Figure 15 (c), a virtual device ID, a serial number of the photographing device 10, image data, and device information are managed in association with an application ID and a group ID that identifies a group that uses the application.

[0121] The app ID “OTHER-APP” is an example of an app developed by another company, and is not fully compatible with the company’s system. As will be described later, the types of information that can be sent and received between the photographing device 10 and the server 90 related to the company’s system are limited.

[0122] 15, the account management table constituting the account management DB 5004 also manages management information used by the communications terminals 90, 90A, and B to manage the photographing device 10 and image identification information. The image identification information is an example of information other than an image or second other information different from the other information, and the management information is an example of the second other information.

[0123] FIG. 16 is a conceptual diagram showing an example of a setting information management table according to the present embodiment.

[0124] FIG. 16A shows a setting information management table constituting the setting information management DB 1001 constructed in the storage unit 1000 of the image capturing device 10 shown in FIG. 13 and the setting information management DB 5001 constructed in the storage unit 5000 of the server 50. As shown in FIG.

[0125] In the setting information management tables constituting the setting information management DB 1001 and the setting information management DB 5001, image processing setting information including imaging apparatus side image processing information and server side image processing information is managed in association with a team ID and a virtual device ID.

[0126] The photographing device-side image processing information is an example of first processing information indicating a first image processing executed on the photographing device 10 side as a transmission source device, and the server-side image processing information is an example of second processing information indicating a second image processing executed on the server 50 side as a transmission destination device. The first image processing executed by the photographing device 10 can be image processing with a relatively low load, and the second image processing executed by the server 50 can be image processing with a relatively high load.

[0127] Note that the first image processing and the second image processing can be applied to the present embodiment as long as they are processing on an image, such as a stitching process for generating a celestial sphere image by stitching together images acquired by each of the image pickup elements 103 a and 103 b described later, zenith correction for adjusting the top and bottom of an image in a vertically upward or downward direction, camera shake correction for correcting image blur caused by camera shake or the like, super-resolution processing for improving image resolution, flare correction for correcting flare that causes part or all of an image to become white due to strong light being reflected by a lens or the like, object detection processing for detecting specific objects such as faces, people, and markers for object detection, and denoising processing for removing or reducing noise generated in an image.

[0128] There are various other types of image processing, but in this embodiment, there may be image processing executed only by the image capture device 10 and the image processing described above as an example. An example of image processing executed only by the image capture device 10 is shading correction and color correction processed by the ISP. Images captured by the image capture device 10 are compressed by the image capture device 10 before being transmitted to the server 50. Therefore, image processing that must be performed before this image compression corresponds to image processing executed only by the image capture device 10. In other words, image processing that can be performed after image compression can be treated as the first image processing or the second image processing.

[0129] That is, the image processing in this embodiment is roughly divided into two types: image processing specific to the photographing device that is executed only by the photographing device 10, and first and second image processing executed on the photographing device side and the server side. Image processing in which the roles are shared between the photographing device side and the server side as described later will be described with the first and second image processing as the subject.

[0130] FIG. 16B shows a setting information management table constituting the setting information management DB 9001 constructed in the storage unit 9000 of the communication terminal 90 shown in FIG. 13 and the setting information management DB 5001 constructed in the storage unit 5000 of the server 50.

[0131] In the setting information management tables constituting the setting information management DB 9001 and the setting information management DB 5001, image processing setting information including communication terminal side processing information and server side image processing information is managed in association with a team ID and a virtual device ID.

[0132] The communication terminal-side image processing information is an example of first processing information indicating a first image processing executed on the communication terminal 90 side as a transmission source device, and the server-side image processing information is an example of second processing information indicating a second image processing executed on the server 50 side as a transmission destination device. The first image processing executed by the communication terminal 90 can be image processing with a relatively low load, and the second image processing executed by the server 50 can be image processing with a relatively high load.

[0133] FIG. 16C shows a setting information management table constituting the setting information management DB 1001 constructed in the storage unit 1000 of the image capturing device 10 shown in FIG.

[0134] In the setting information management tables constituting the setting information management DB1001 and the setting information management DB9001, image processing setting information including photographing device side processing information and communication terminal side image processing information is managed in association with a team ID and a virtual device ID.

[0135] The photographing device-side image processing information is an example of first processing information indicating a first image processing executed on the photographing device 10 side, and the communication terminal-side image processing information is an example of second processing information indicating a second image processing executed on the communication terminal 90 side. The first image processing executed by the photographing device 10 can be image processing with a relatively low load, and the second image processing executed by the communication terminal 90 can be image processing with a relatively high load.

[0136] The image processing setting information in the setting information management table shown in FIG. 16(a) is set in association with a transmission path for directly transmitting an image from the photographing device 10 to the server 50.

[0137] 16(b) and 16(c) is set in association with a transmission path for transmitting an image from the photographing device 10 to the server 50 via the communication terminal 90. In this case, the image processing setting information may include all of the photographing device side processing information, the communication terminal side image processing information, and the server side image processing information.

[0138] Fig. 17 is a conceptual diagram showing an example of a device information management table according to the present embodiment. Fig. 17 shows a device information management table constituting the device information management DB 1003 constructed in the storage unit 1000 of the image capturing device 10 shown in Fig. 13.

[0139] 17, the type of transmittable device information is managed depending on whether the destination is the server 50 or the communication terminal 90. When the destination is the communication terminal 90, the type of transmittable device information is managed in association with the application ID described in FIG.

[0140] FIG. 18 is a sequence diagram showing an example of a registration process for server direct transmission according to the present embodiment.

[0141] The reception unit 92 of the communication terminal 90 receives the input information when the user inputs a user ID and password (step S1), and the authentication unit 97 authenticates the information received by the reception unit 92 (step S2).

[0142] The transmitting / receiving unit 91 transmits the user ID authenticated in step S2 to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the user ID transmitted from the communication terminal 90 (step S3).

[0143] Authentication unit 53 refers to account management DB 5004 via storage / readout unit 59, and authenticates the user ID received in step S3 (step S4).

[0144] The reception unit 92 of the communication terminal 90 receives information related to the registration request when the user inputs information requesting registration of a transmission path for direct transmission from the photographing device 10 to the server 50, and the memory / reading unit 99 reads out the team ID stored in the memory unit 9000 in correspondence with the user ID (step S5).

[0145] The transmitter / receiver unit 91 transmits the information indicating the transmission route received in step S5 and the team ID read out in step S5 to the photographing device 10, the transmitter / receiver unit 11 of the photographing device 10 receives the information indicating the transmission route and the team ID sent from the communication terminal 90, and the data management unit 17 stores the received information indicating the transmission route in the memory unit 1000 via the memory / readout unit 19 (step S6).

[0146] The data management unit 17 reads the app ID stored in correspondence with the communication terminal 90 and the serial number of the photographing device 10 from the memory unit 1000 using the memory / read unit 19, and the transceiver unit 11 transmits the app ID, serial number and team ID to the server 50, and the transceiver unit 51 of the server 50 receives the app ID, serial number and team ID transmitted from the photographing device 10 (step S7).

[0147] The authentication unit 53 refers to the account management DB 5004 to check whether the various information received in step S7 has already been registered, and if it has not been registered, assigns a virtual device ID corresponding to the serial number (step S8). The data processing unit 54 causes the storage / reading unit 59 to associate the various information received in step S7 with the virtual device ID and store them in the account management DB 5004 (step S9).

[0148] The transmitting / receiving unit 51 transmits the virtual device ID assigned in step S8 to the photographing device 10, and the transmitting / receiving unit 11 of the photographing device 10 receives the virtual device ID transmitted from the server 50 (step S10). As a result, a transmission route for direct transmission from the photographing device 10 to the server 50, i.e., a second transmission route not passing through the communication terminal 90, is registered.

[0149] If, in step S8, it is confirmed that the various information received in step S7 has been registered, in step S10, the transceiver unit 51 transmits an error message indicating that the information has been registered to the photographing device 10, and the transceiver unit 11 of the photographing device 10 receives the error message transmitted from the server 50.

[0150] FIG. 19A is a sequence diagram showing an example of direct transmission of an image to a server according to this embodiment.

[0151] The data management unit 17 of the photographing device 10 reads the image from the image management DB 1002 using the memory / read unit 19, and the transmission / reception unit 11 transmits the application ID, virtual device ID, image data representing the image, and image identification information to the server 50, and the transmission / reception unit 51 of the server 50 receives the application ID, virtual device ID, image data, and image identification information transmitted from the photographing device 10 (step S21).

[0152] The authentication unit 53 authenticates the various information received in step S21 by referring to the account management DB 5004 (step S22), and the data processing unit 54 causes the storage / reading unit 59 to store the image related to the image data received in step S21 in the image management DB 5002, and also causes the image identification information and the path to the image management DB 5002 to be stored in the account management DB 5004 in association with the application ID and virtual device ID received in step S21 (step S23).

[0153] If image processing is to be performed on the image corresponding to the image data received in step S21, in step S23, the data processing unit 54 causes the storage / reading unit 59 to store the processed image in the image management DB 5002.

[0154] FIG. 19B is a sequence diagram showing an example of direct transmission of device information to the server according to this embodiment.

[0155] The data management unit 17 of the photographing device 10 uses the memory / read unit 19 to refer to the device information management table shown in FIG. 17 and reads out device information that can be received by the server 50 from the device information management DB 1003, and the transmission / reception unit 11 transmits the application ID, virtual device ID, and device information read out from the memory unit 1000 to the server 50, and the transmission / reception unit 51 of the server 50 receives the application ID, virtual device ID, and device information transmitted from the photographing device 10 (step S31).

[0156] The authentication unit 53 authenticates the various information received in step S31 by referring to the account management DB 5004 (step S32), and the data processing unit 54 causes the storage / reading unit 59 to store the device information received in step S31 in the device information management DB 5003, and also causes the account management DB 5004 to store a path to the device information management DB 5003 in association with the application ID and virtual device ID received in step S31 (step S33).

[0157] FIG. 20 is a sequence diagram showing an example of a registration process for transmission and reception mediation according to the present embodiment.

[0158] The reception unit 92 of the communication terminal 90 receives the input information when the user inputs a user ID and password (step S41), and the authentication unit 97 authenticates the information received by the reception unit 92 (step S42).

[0159] The transmitting / receiving unit 91 transmits the user ID authenticated in step S42 to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the user ID transmitted from the communication terminal 90 (step S43).

[0160] The authentication unit 53 refers to the account management DB 5004 by the storage / readout unit 59, and authenticates the user ID received in step S43 (step S44).

[0161] The reception unit 92 of the communication terminal 90 receives information related to the registration request when the user inputs information requesting registration of a transmission path for transmission from the photographing device 10 to the server 50 via the communication terminal 90, and the memory / reading unit 99 reads out the team ID stored in the memory unit 9000 in correspondence with the user ID (step S45).

[0162] The transmitter / receiver unit 91 transmits the information indicating the transmission route received in step S45 and a request to obtain the serial number of the photographing device 10 to the photographing device 10, and the transmitter / receiver unit 11 of the photographing device 10 receives the information indicating the transmission route and the request transmitted from the communication terminal 90, and the data management unit 17 stores the received information indicating the transmission route in the memory unit 1000 via the memory / readout unit 19 (step S46).

[0163] The data management unit 17 reads the serial number of the photographing device 10 from the memory unit 1000 using the memory / reading unit 19, and the transceiver unit 11 transmits the serial number to the communication terminal 90, and the transceiver unit 91 of the communication terminal 90 receives the serial number transmitted from the photographing device 10 (step S47).

[0164] The transmission / reception unit 91 transmits the application ID, the serial number received in step S47, the team ID read in step S45, and the user ID to the server 50, and the transmission / reception unit 51 of the server 50 receives the application ID, serial number, team ID, and user ID transmitted from the communication terminal 90 (step S48).

[0165] The authentication unit 53 refers to the account management DB 5004 to check whether the various information received in step S48 has already been registered, and if it has not been registered, assigns a virtual device ID corresponding to the serial number (step S49).The data processing unit 54 causes the storage / reading unit 59 to associate the various information received in step S48 with the virtual device ID and store them in the account management DB 5004 (step S50).

[0166] The transmitting / receiving unit 51 transmits the virtual device ID assigned in step S49 to the photographing device 10, and the transmitting / receiving unit 11 of the photographing device 10 receives the virtual device ID transmitted from the server 50 (step S51). As a result, a first transmission path for transmission from the photographing device 10 to the server 50 via the communication terminal 90 is registered.

[0167] If, in step S49, it is confirmed that the various information received in step S48 has been registered, in step S51, the transceiver unit 51 transmits an error message indicating that the information has been registered to the photographing device 10, and the transceiver unit 11 of the photographing device 10 receives the error message transmitted from the server 50.

[0168] FIG. 21 is a sequence diagram showing an example of image transmission / reception mediation processing according to this embodiment.

[0169] The reception unit 92 of the communication terminal 90 receives the input information when the user inputs a user ID and password (step S61), and the authentication unit 97 authenticates the information received by the reception unit 92 (step S62).

[0170] The transmitting / receiving unit 91 transmits the user ID authenticated in step S62 to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the user ID transmitted from the communication terminal 90 (step S63).

[0171] The authentication unit 53 refers to the account management DB 5004 via the storage / readout unit 59, and authenticates the user ID received in step S63 (step S64).

[0172] The receiving unit 92 of the communication terminal 90 receives information related to the upload request when the user inputs information requesting uploading of an image (step S45).

[0173] The transmitting / receiving unit 91 transmits an image acquisition request to the photographing device 10, and the transmitting / receiving unit 11 of the photographing device 10 receives the acquisition request transmitted from the communication terminal 90 (step S66).

[0174] The data management unit 17 reads the image from the image management DB 1002 using the memory / read unit 19, and the transmission / reception unit 11 transmits the virtual device ID, image data indicating the read image, and image identification information to the communication terminal 90, and the transmission / reception unit 91 of the communication terminal 90 receives the virtual device ID, the image data, and the image identification information (step S67).

[0175] The data control unit 95 reads out the management information of the photographing device 10 from the memory unit 9000 using the memory / reading unit 99, and the transmission / reception unit 91 transmits the application ID, virtual device ID, user ID, image data, image identification information, and management information to the server 50, and the transmission / reception unit 51 of the server 50 receives the application ID, virtual device ID, user ID, image data, image identification information, and management information transmitted from the communication terminal 90 (step S68).

[0176] The authentication unit 53 authenticates the various information received in step S68 by referring to the account management DB 5004 (step S69), and the data processing unit 54 causes the storage / reading unit 59 to store the image related to the image data received in step S68 in the image management DB 5002, and also stores the management information, image identification information, and a path to the image management DB 5002 in association with the application ID and virtual device ID received in step S68 in the account management DB 5004 (step S70).

[0177] If image processing is to be performed on the image corresponding to the image data received in step S68, then in step S70, the data processing unit 54 causes the storage / reading unit 59 to store the processed image in the image management DB 5002.

[0178] FIG. 22 is a sequence diagram showing an example of a transmission / reception mediation process for device information according to the present embodiment.

[0179] The reception unit 92 of the communication terminal 90 receives the input information when the user inputs a user ID and password (step S81), and the authentication unit 97 authenticates the information received by the reception unit 92 (step S82).

[0180] The transmitting / receiving unit 91 transmits the user ID authenticated in step S82 to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the user ID transmitted from the communication terminal 90 (step S83).

[0181] The authentication unit 53 refers to the account management DB 5004 by the storage / readout unit 59, and authenticates the user ID received in step S83 (step S84).

[0182] The receiving unit 92 of the communication terminal 90 receives information related to the upload request when the user inputs information requesting uploading of the device information (step S85).

[0183] The transmitting / receiving unit 91 transmits a request for obtaining the device information to the photographing device 10, and the transmitting / receiving unit 11 of the photographing device 10 receives the request transmitted from the communication terminal 90 (step S86).

[0184] The data management unit 17, using the memory / read unit 19, refers to the device information management table shown in FIG. 17 and reads from the memory unit 1000 device information that can be received by the communication terminal 90. The transmission / reception unit 11 transmits the virtual device ID and the device information read from the memory unit 1000 to the communication terminal 90, and the transmission / reception unit 91 of the communication terminal 90 receives the virtual device ID and device information transmitted from the photographing device 10 (step S87).

[0185] The data control unit 95 reads out the management information of the photographing device 10 from the memory unit 9000 using the memory / reading unit 99, and the transmission / reception unit 91 transmits the application ID, virtual device ID, user ID, device information, and management information to the server 50, and the transmission / reception unit 51 of the server 50 receives the application ID, virtual device ID, device information, and management information transmitted from the communication terminal 90 (step S88).

[0186] The authentication unit 53 authenticates the various information received in step S88 by referring to the account management DB 5004 (step S89), and the data processing unit 54 causes the storage / reading unit 59 to store the device information received in step S88 in the device information management DB 5003, and also causes the account management DB 5004 to store the path to the device information management DB 5003 in association with the management information, the application ID and virtual device ID received in step S88 (step S90).

[0187] FIG. 23 is a sequence diagram showing an example of another transmission / reception mediation registration process according to the present embodiment.

[0188] The reception unit 92 of the communication terminal 90 receives information related to the registration request when the user inputs information requesting registration of a transmission path for transmission from the photographing device 10 to the server 50 via the communication terminals 90A and 90B, and the memory / reading unit 99 reads out the team ID stored in the memory unit 9000 in correspondence with the user ID (step S101).

[0189] The transmitter / receiver unit 91 transmits to the photographing device 10 the information indicating the transmission route received in step S101 and a request to obtain the serial number of the photographing device 10, the transmitter / receiver unit 11 of the photographing device 10 receives the information indicating the transmission route and the request to obtain sent from the communication terminal 90, and the data management unit 17 causes the memory / read unit 19 to store the received information indicating the transmission route in the memory unit 1000 (step S102).

[0190] The data management unit 17 reads the serial number of the photographing device 10 from the memory unit 1000 using the memory / reading unit 19, and the transceiver unit 11 transmits the serial number to the communication terminal 90, and the transceiver units 91 of the communication terminals 90A and 90B receive the serial number transmitted from the photographing device 10 (step S103).

[0191] The transmitter / receiver 91 transmits the application ID, the serial number received in step S103, and the group ID to the server 50, and the transmitter / receiver 51 of the server 50 receives the application ID, the serial number, and the group ID transmitted from the communication terminals 90A and 90B (step S104).

[0192] The authentication unit 53 refers to the account management DB 5004 to check whether the various information received in step S104 has already been registered, and if it has not been registered, assigns a virtual device ID corresponding to the serial number (step S105).The data processing unit 54 causes the storage / reading unit 59 to associate the various information received in step S104 with the virtual device ID and store them in the account management DB 5004 (step S106).

[0193] The transmitting / receiving unit 51 transmits the virtual device ID assigned in step S106 to the photographing device 10, and the transmitting / receiving unit 11 of the photographing device 10 receives the virtual device ID transmitted from the server 50 (step S107).

[0194] If, in step S105, it is confirmed that the various information received in step S104 has been registered, in step S107, the transceiver unit 51 transmits an error message indicating that the information has been registered to the photographing device 10, and the transceiver unit 11 of the photographing device 10 receives the error message transmitted from the server 50.

[0195] FIG. 24 is a sequence diagram showing an example of another image transmission / reception mediation process according to the present embodiment.

[0196] The receiving unit 92 of the communication terminals 90A and 90B receives information related to an upload request when a user inputs information requesting uploading of an image (step S111).

[0197] The transmitting / receiving unit 91 transmits an image acquisition request to the photographing device 10, and the transmitting / receiving unit 11 of the photographing device 10 receives the acquisition request transmitted from the communication terminal 90 (step S112).

[0198] Data management unit 17 reads the image from image management DB 1002 using storage / read unit 19, and transmission / reception unit 11 transmits the virtual device ID and image data indicating the read image to communications terminals 90A and 90B, and transmission / reception units 91 of communications terminals 90A and 90B receive the virtual device ID and image data (step S113). Data management unit 17 determines that the image identification information cannot be received by the applications related to transmission / reception provided in communications terminals 90A and 90B, and does not cause transmission / reception unit 11 to transmit the image identification information to communications terminals 90A and 90B, unlike step 67 in Fig. 21.

[0199] The transmitting / receiving unit 91 transmits the application ID, the virtual device ID, the group ID, and the image data to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the application ID, the virtual device ID, the group ID, and the image data transmitted from the communication terminals 90A and 90B (step S114). Based on the application related to the transmission and reception, the data control unit 95 does not cause the transmitting / receiving unit 51 to transmit management information to the server 50, unlike step 68 in FIG.

[0200] The authentication unit 53 authenticates the various information received in step S114 by referring to the account management DB 5004 (step S115), and the data processing unit 54 causes the storage / reading unit 59 to store the image related to the image data received in step S114 in the image management DB 5002, and also causes the account management DB 5004 to store the path to the image management DB 5002 in association with the application ID and virtual device ID received in step S114 (step S116).

[0201] If image processing is to be performed on the image corresponding to the image data received in step S114, then in step S116, the data processing unit 54 causes the storage / reading unit 59 to store the processed image in the image management DB 5002.

[0202] FIG. 25 is a sequence diagram showing an example of another transmission / reception mediation process of device information according to the present embodiment.

[0203] The receiving unit 92 of the communication terminals 90A and 90B receives information related to an upload request when the user inputs information requesting uploading of device information (step S121).

[0204] The transmitting / receiving unit 91 transmits a request for obtaining the device information to the photographing device 10, and the transmitting / receiving unit 11 of the photographing device 10 receives the request transmitted from the communication terminals 90A and 90B (step S122).

[0205] The data management unit 17, using the memory / read unit 19, refers to the device information management table shown in FIG. 17 and reads from the memory unit 1000 device information that can be received by the communication terminals 90A, 90B, and the transmission / reception unit 11 transmits the virtual device ID and the device information read from the memory unit 1000 to the communication terminals 90A, 90B, and the transmission / reception units 91 of the communication terminals 90A, 90B receive the virtual device ID and device information transmitted from the photographing device 10 (step S123).

[0206] The transmitting / receiving unit 91 transmits the application ID, the virtual device ID, the user ID, and the device information to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the application ID, the virtual device ID, and the device information transmitted from the communication terminals 90A and 90B (step S124). Based on the application related to the transmission / reception, the data control unit 95 does not cause the transmitting / receiving unit 91 to transmit the management information to the server 50, unlike step 88 in FIG.

[0207] The authentication unit 53 authenticates the various information received in step S124 by referring to the account management DB 5004 (step S125), and the data processing unit 54 causes the storage / reading unit 59 to store the device information received in step S124 in the device information management DB 5003, and also causes the account management DB 5004 to store a path to the device information management DB 5003 in association with the application ID and virtual device ID received in step S124 (step S126).

[0208] FIG. 26 is a sequence diagram showing an example of the device information display process according to the present embodiment.

[0209] The reception unit 92 of the communication terminal 90 receives the input information when the user inputs a user ID and password (step S131), and the authentication unit 97 authenticates the information received by the reception unit 92 (step S132).

[0210] The transmitting / receiving unit 91 transmits the user ID authenticated in step S132 to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the user ID transmitted from the communication terminal 90 (step S133).

[0211] The authentication unit 53 refers to the account management DB 5004 by the storage / readout unit 59, and authenticates the user ID received in step S133 (step S134).

[0212] The receiving unit 92 of the communication terminal 90 receives information related to the display request when the user inputs information requesting display of the device information of the target image capturing device 10 (step S135).

[0213] The transmission / reception unit 91 transmits the application ID, the user ID, and the serial number of the photographing device 10 displaying the equipment information to the server 50, and the transmission / reception unit 51 of the server 50 receives the application ID, the user ID, and the serial number transmitted from the communication terminal 90 (step S136).

[0214] The authentication unit 53 authenticates the various information received in step S136 by referring to the account management DB 5004 (step S137), and the data processing unit 54 reads, via the storage / reading unit 59, the device information and management information of the photographing device 10 associated with the serial number received in step S136 from the device information management DB 5003 and the account management DB 5004 (step S138).

[0215] The transmitter / receiver 51 transmits the device information and management information read in step S138 to the communication terminal 90, the transmitter / receiver 91 of the communication terminal 90 receives the device information and management information transmitted from the server 50 (step S139), and the display control unit 93 displays the device information and management information received in step S138 on the display 906 (step S140).

[0216] FIG. 27 is a sequence diagram showing another example of the device information display process according to the present embodiment.

[0217] The receiving unit 92 of the communication terminal 90A receives information related to the display request when the user inputs information requesting display of the device information of the target photographing device 10 (step S151).

[0218] The transceiver unit 91 transmits the application ID, group ID, and serial number of the photographing device 10 displaying the equipment information to the server 50, and the transceiver unit 51 of the server 50 receives the application ID, group ID, and serial number transmitted from the communication terminal 90 (step S152).

[0219] The authentication unit 53 authenticates the various information received in step S152 by referring to the account management DB 5004 (step S137), and the data processing unit 54 reads, via the storage / reading unit 59, the device information and management information of the photographing device 10 associated with the serial number received in step S152 from the device information management DB 5003 and the account management DB 5004 (step S154).

[0220] The transmitter / receiver 51 transmits the device information and management information read in step S154 to the communication terminal 90, the transmitter / receiver 91 of the communication terminal 90 receives the device information and management information transmitted from the server 50 (step S155), and the display control unit 93 displays the device information and management information received in step S155 on the display 906 (step S156).

[0221] FIG. 28 is a flowchart showing an example of an image transmission process in the imaging device according to this embodiment.

[0222] The shooting control unit 13 captures a still image or a video using the still image shooting unit 15 or the video shooting unit 14 when a predetermined condition is triggered, and the storage / reading unit 19 stores a captured image representing the captured still image or video in the image management DB 1002 (step S161).

[0223] The data management unit 17 checks the information indicating the transmission route stored in the memory unit 1000 by the memory / read unit 19 to determine whether a transmission route for direct transmission to the server 50 is registered (step S162), determines whether image processing is necessary based on the image processing setting information stored in the setting information management DB 1001, and determines whether image processing is possible based on the load information of the CPU 111, etc. (step S163).

[0224] If image processing is neither necessary nor possible, the transmitting / receiving unit 11 transmits the captured image captured in step S161 and image identification information indicating that it is a captured image to be transmitted to the server 50 (step S164).

[0225] If image processing is necessary and possible, the image processing unit 16 reads out the image processing information of the shooting device side from the setting information management DB 1001 shown in Figure 16 (a) using the memory / readout unit 19, and performs the image processing indicated in the image processing information of the shooting device side on the still image or video captured in step S161, and the memory / readout unit 19 stores the processed image indicating the still image or video processed by the image processing unit 16 in the image management DB 1002 in correspondence with the captured image (step S165).

[0226] Here, the photographing apparatus side image processing information in the setting information management DB 1001 shown in FIG. 16(a) is set in association with a transmission path for direct transmission to the server 50.

[0227] Next, the transmitting / receiving unit 11 transmits to the server 50 the processed image that has been subjected to the image processing in step S165, and image identification information indicating that it is a processed image to be transmitted and indicating the image processing performed in step S165 (step S166).

[0228] If it is determined in step S162 that a transmission path for direct transmission to server 50 is not registered, data management unit 17 determines whether transmission / reception unit 11 has received an image acquisition request from communication terminal 90 (step S167).

[0229] When an image acquisition request is received, the data management unit 17 determines whether image processing is necessary based on the image processing setting information stored in the setting information management DB 1001, and determines whether image processing is possible based on the load information of the CPU 111, etc. (step S168).

[0230] If image processing is neither necessary nor possible, the transmitting / receiving unit 11 transmits the captured image captured in step S161 and image identification information indicating that it is the captured image to be transmitted to the communication terminal 90 (step S169).

[0231] If image processing is necessary and possible, the image processing unit 16 reads out the image processing information of the shooting device side from the setting information management DB 1001 shown in Figure 16 (c) using the memory / readout unit 19, and performs the image processing indicated in the image processing information of the shooting device side on the still image or video captured in step S161, and the memory / readout unit 19 stores the processed image indicating the still image or video processed by the image processing unit 16 in the image management DB 1002 in correspondence with the captured image (step S170).

[0232] Here, the photographing apparatus side image processing information in the setting information management DB 1001 shown in FIG. 16C is set in association with a transmission path for transmission to the server 50 via the communication terminal 90.

[0233] Next, the transmitting / receiving unit 11 transmits to the communication terminal 90 the processed image that has been subjected to the image processing in step S170, and image identification information indicating that it is a processed image to be transmitted and indicating the image processing performed in step S170 (step S171).

[0234] FIG. 29(a) is a flowchart showing an example of an image transmission / reception process in the communication terminal according to this embodiment.

[0235] The data control unit 95 of the communication terminal 90 determines whether the transmitting / receiving unit 91 has received an image from the photographing device 10 (step S181).

[0236] When an image is received, the data control unit 95 determines whether image processing is necessary based on the image processing setting information stored in the setting information management DB 1001 and the image identification information received along with the image in step S181, and determines whether image processing is possible based on load information of the CPU 111, etc. (step S182).

[0237] If image processing is neither necessary nor possible, the transmission / reception unit 91 adds management information of the photographing device 10 read from the memory unit 9000 by the memory / reading unit 99 to the image received in step S181 and the image identification information received together with the image, and transmits the resulting image to the server 50 (step S183).

[0238] If image processing is necessary and possible, the image processing unit 94 reads out the communications terminal side image processing information from the setting information management DB9001 shown in Figure 16 (b) or (c) using the memory / readout unit 19, and performs image processing on the image received in step S181 based on the communications terminal side image processing information and the image identification information received in step S181, and the data control unit 95 causes the memory / readout unit 99 to store the processed image that has been processed by the image processing unit 94 in the image management DB9002 (step S184).

[0239] Specifically, if the image identification information received in step S181 matches the photographing device side image processing information shown in FIG. 16(c), the image processing unit 94 executes the communication terminal side image processing shown in FIG. 16(c).

[0240] On the other hand, if the image identification information received in step S181 does not match the image processing information on the photographing device side shown in Figure 16(c), the image processing unit 94 determines that the photographing device 10 was unable to perform all of the image processing that it should have performed, and performs the image processing that the photographing device 10 was unable to perform and the communication terminal side image processing shown in Figure 16(c).

[0241] Next, the transceiver unit 11 adds management information of the photographing device 10 to the processed image on which image processing was performed in step S184 and image identification information indicating that the image to be transmitted is a processed image and the image processing performed in step S184, as in step S183, and transmits the resulting image to the server 50 (step S185).

[0242] FIG. 29B is a flowchart showing an example of an image receiving process in the server according to this embodiment.

[0243] The data control unit 95 of the communication terminal 90 determines whether the transmitting / receiving unit 91 has received an image from the photographing device 10 (step S191).

[0244] When an image is received, the data control unit 95 determines whether image processing is necessary based on the image processing setting information stored in the setting information management DB 1001 and the image identification information received along with the image in step S181, and determines whether image processing is possible based on load information of the CPU 111, etc. (step S192).

[0245] If image processing is neither necessary nor possible, the transmission / reception unit 91 adds management information of the photographing device 10 read from the memory unit 9000 by the memory / reading unit 99 to the image received in step S191 and the image identification information received together with the image, and transmits the image to the server 50 (step S193).

[0246] If image processing is necessary and possible, the image processing unit 94 reads out communications terminal side image processing information from the setting information management DB9001 shown in Figure 16 (b) or (c) using the memory / reading unit 99, and performs image processing on the image received in step S191 based on the communications terminal side image processing information and the image identification information received in step S191, and the data control unit 95 causes the memory / reading unit 99 to store the processed image that has been processed by the image processing unit 94 in the image management DB9002 (step S194).

[0247] Specifically, if the image identification information received in step S191 matches the photographing device side image processing information shown in FIG. 16(c), the image processing unit 94 executes the communication terminal side image processing shown in FIG. 16(c).

[0248] On the other hand, if the image identification information received in step S191 does not match the image processing information on the photographing device side shown in Figure 16(c), the image processing unit 94 determines that the photographing device 10 was unable to perform all of the image processing that it should have performed, and performs the image processing that the photographing device 10 was unable to perform and the communication terminal side image processing shown in Figure 16(c).

[0249] FIG. 30 is a flowchart showing an example of device information transmission processing in the imaging device according to this embodiment.

[0250] The data management unit 17 checks the information indicating the transmission route stored in the memory unit 1000 by the memory / read unit 19 to determine whether a transmission route for direct transmission to the server 50 has been registered (step S201), and determines whether a specified transmission timing has arrived, such as when the imaging device 10 is started up or periodically (step S202).

[0251] When the transmission timing arrives, the data management unit 17 reads out device information that can be received by the server 50 from the device information management DB 1003 using the memory / read unit 19, and the transmitter / receiver 11 transmits the device information read out from the memory unit 1000 to the server 50 together with the application ID and virtual device ID (step S203).

[0252] If it is determined in step S201 that a transmission path for direct transmission to server 50 is not registered, data management unit 17 determines whether transmission / reception unit 11 has received a request to obtain device information from communication terminal 90 (step S204).

[0253] When an acquisition request is received, the data management unit 17 reads, via the memory / reading unit 19, from the memory unit 1000, device information that can be received by the communication terminal 90, and the transmission / reception unit 11 transmits the device information read from the memory unit 1000 together with the virtual device ID to the communication terminal 90 (step S205).

[0254] FIG. 31 is a diagram showing an example of a display screen for device information according to this embodiment, and shows a display screen 2000 that the display control unit 93 of the communication terminal 90, 90A causes to be displayed on the display 906 in step 140 in FIG. 26 or step 156 in FIG. 27.

[0255] 31(a), the display control unit 93 displays a map 2140, a position 2042 on the map based on the position information of the image capturing device 10, and a serial number 2045 on the display screen 2000. The position information of the image capturing device 10 and the serial number are included in the device information.

[0256] 31(b), the display control unit 93 includes a time series graph 2051 showing the number of error occurrences based on error information of the image capturing device 10, a time series graph 2053 showing temperature changes based on temperature information of the board, a time series graph 2055 showing temperature changes based on temperature information of the battery, a time series graph 2057 showing remaining charge changes based on remaining battery charge information, a time series graph 2059 showing remaining charge changes based on remaining storage charge information, a time series graph 2061 showing the number of ON / OFF times based on power ON / OFF information of the image capturing device 10, and a time series graph 2063 showing the number of uploads based on the number of image transmissions. The error information of the image capturing device 10, the temperature information of the board, the temperature information of the battery, the remaining battery charge information, the remaining storage charge information, the power ON / OFF information of the image capturing device 10, and the number of image transmissions are included in the device information.

[0257] Here, even if all of the information shown in Figures 31(a) and (b) is displayed on the display screen 2000 of a communication terminal 90 equipped with an application related to transmission and reception developed by the company, some of the information shown in Figures 31(a) and (b) may not be displayed on the display screen 2000 of a communication terminal 90A equipped with an application related to transmission and reception developed by another company, due to the inability to transmit and receive all of the device information.

[0258] FIG. 32 is a diagram showing an example of a display screen for management information according to this embodiment, and shows a display screen 2000 that the display control unit 93 of the communication terminal 90, 90A causes to be displayed on the display 906 in step 140 in FIG. 26 or step 156 in FIG. 27.

[0259] The display control unit 93 displays, on the display screen 2000, the number of members 2010 based on the user ID and team ID acquired from the account management DB 5004 and a time series graph 2015 showing the change in the number of members, the number of devices (imaging devices) 2020 based on the virtual device ID acquired from the account management DB 5004 and a time series graph 2025 showing the change in the number of devices, and the number of stored images 2030 acquired from the image management DB 5002 and a time series graph 2035 showing the change in the number of images. The user ID and team ID, the virtual device ID and the number of stored images are included in the management information.

[0260] Here, even if all of the information shown in FIG. 32 is displayed on display screen 2000 in communication terminal 90 equipped with an application related to transmission and reception developed by the company, communication terminal 90A equipped with an application related to transmission and reception developed by another company does not treat the team ID as management information, and therefore, of the information shown in FIG. 32, at least number of members 2010 and time series graph 2015 are not displayed on display screen 2000.

[0261] FIG. 33 is a diagram showing another example of a display screen for management information according to this embodiment, and shows a display screen 2000 that the display control unit 93 of the communication terminal 90, 90A causes to be displayed on the display 906 in step 140 in FIG. 26 or step 156 in FIG. 27.

[0262] The display control unit 93 displays on the display screen 2000, as information of a specific app based on the app ID obtained from the account management DB 5004, a time series graph 2071 showing the change in the number of image uploads based on the number of image data obtained from the account management DB 5004, a time series graph 2073 showing the change in the number of photographs taken based on the number of photographs taken obtained from the device information management DB, a time series graph 2075 showing the change in the number of connections based on the number of connections between the app and the photographing device obtained from the device information management DB, and a time series graph 2077 showing the change in the number of logins based on the number of logins to the communication terminals 90 and 90A as management information.

[0263] The information shown in FIG. 33 is different for each application used in the communication terminals 90 and 90A.

[0264] As described above, in one embodiment of the present invention, the imaging device 10 or communication terminal 90, which is an example of a source device, and the server 50, which is an example of a destination device, are configured to be usable by, for example, real estate agents who manage or sell real estate properties, real estate agencies that introduce real estate properties, and construction companies that manage structures such as buildings, but the application of the present invention is not limited to these.

[0265] Digital twins, which link the real world with virtual space, are new technologies that utilize the Internet of Things (IoT), in which all things connect to the Internet and communicate, artificial intelligence (AI), Augmented Reality (AR), which provides information to and expands the real world, and Virtual Reality (VR), which makes virtual space look like the real world.Digital twins make it possible to collect and analyze data in real time using actual sensors, evaluate risks, and identify the causes of failures and errors, and so digital twins are beginning to be used primarily in the manufacturing industry, such as in automobiles and production equipment in factories.

[0266] In another embodiment, the functional configuration of the imaging device described in the above-mentioned embodiment may be a configuration that is possessed by a moving body, production equipment in a factory, etc. Here, the moving body may be a vehicle, an aircraft, a ship, a mobile robot, etc.

[0267] FIG. 34 is a diagram showing an example of the overall configuration of an image processing system according to a modified embodiment of the present invention.

[0268] The image processing system 1 shown in FIG. 34 includes an automobile 70, which is an example of a moving body, and a server 50. The automobile 70 includes an imaging device 10 serving as a drive recorder, a control device 70A, and a sensor data storage unit 70B.

[0269] The server 50 and the control device 70A of the automobile 70, which constitute the image processing system 1, can communicate with each other via the communication network 100. Furthermore, the image capturing device 10 and the sensor data storage unit 70B can each communicate with the control device 70A.

[0270] That is, the image capturing device 10 can communicate with the server 50 via the control device 70A and the communication network 100. The sensor data storage unit 70B is a modified example of the device information management DB 1003 shown in Fig. 13, and stores and manages data acquired by various sensors of the automobile.

[0271] The image processing system 1 shown in FIG. 34 directly transmits data acquired by various sensors of an automobile 70 and images captured by an image capturing device 10 serving as a drive recorder to a server 50 of an automobile manufacturer.

[0272] An automobile owner has a contract with an automobile manufacturer for a service in which data relating to the driving of the automobile 70 acquired by various sensors and video footage captured by a drive recorder are uploaded to the automobile manufacturer's server 50, and the automobile manufacturer analyzes the data and video footage and provides the owner with various analysis results, such as the owner's driving safety level, notifications for maintenance and part replacement, etc. An automobile 70 that has contracted for this service has a transmission path set up to directly transmit the acquired data and captured video footage from the automobile's wireless communication module to the automobile manufacturer's server 50.

[0273] FIG. 35 is a diagram showing an example of the overall configuration of an image processing system according to a second modified example of the embodiment of the present invention.

[0274] The image processing system 1 shown in FIG. 35 includes a communication terminal 90, an automobile 70 which is an example of a moving body, and a server 50. The automobile 70 includes an imaging device 10 as a drive recorder, a control device 70A, and a sensor data storage unit 70B.

[0275] The server 50 and the communication terminal 90 constituting the image processing system 1 can communicate with each other via a communication network 100. Furthermore, the photographing device 10 and the sensor data storage unit 70B can each communicate with the communication terminal 90 via the control device 70A.

[0276] That is, the image capturing device 10 can communicate with the server 50 via the control device 70A, the communication terminal 90, and the communication network 100. The sensor data storage unit 70B is a modified example of the device information management DB 1003 shown in Fig. 13, and stores and manages data acquired by various sensors of the automobile.

[0277] The image processing system 1 shown in FIG. 35 transmits data acquired by various sensors of an automobile 70 and images captured by an image capturing device 10 serving as a drive recorder to a server 50 of an insurance company via a communication terminal 90.

[0278] The owner of the automobile 70 contracts automobile insurance with an insurance company, and in the event of an accident involving the automobile 70, promptly notifies the insurance company of the circumstances of the accident. At that time, the owner starts an app of the insurance company installed on the communication terminal 90, and the app acquires data related to the automobile 70 at the time of the accident (e.g., speed, timing of braking, steering operation, damaged parts, etc.) from the automobile 70, as well as video footage of the accident captured by a drive recorder, and uploads the data and video to the insurance company's server 50, enabling the insurance company to grasp the circumstances of the accident from the data and video, and to obtain evidence necessary for subsequent negotiations with the other party, such as offsetting fault.

[0279] In this way, when uploading data from various sensors and captured video in real time, it is possible to selectively use a format in which the data to be uploaded and the video are sent directly to the server 50, and a format in which the data to be uploaded and the video are sent via the communication terminal 90 only when necessary.

[0280] In yet another embodiment, the present invention may be applied to a surveillance camera system that monitors, for example, a bank, a retail store where a retailer sells products, roads and urban areas managed by a local government, etc. In this case, taking Fig. 1 as an example, the imaging device 10 is installed at a predetermined monitoring position such as a ceiling or wall in a monitored object such as a bank or a retail store.

[0281] When roads or urban areas are to be monitored, the image capturing device 10 is installed on structures such as pedestrian bridges and utility poles. The server 50 may be owned by an image management company that manages images, or may be owned by a bank, a retail store, or a local government.

[0282] The communication terminal 90 may be owned by a bank, a retail store, or a local government, or may be owned by a security company that provides security for the monitored object, or may be owned by both.

[0283] In the case of an application for monitoring a surveillance target, real-time (immediate) sharing of captured images may be required. In this case, the image capturing device 10 performs image processing with a relatively low load at a minimum level that allows the user to check the image, and shares the image with the server 50.

[0284] Image processing that has a relatively high load, such as object detection processing, is executed in the server 50. In this way, the image capturing device 10 can quickly share images with the server 50, and when applied to a surveillance camera system, captured images can be efficiently utilized.

[0285] ●Summary● [First aspect] As described above, the photographing device 10 according to one embodiment of the present invention includes a transmission / reception unit 11 that transmits to the server 50 at least one of an image photographed by the photographing device 10 and information other than the image via a first transmission path that passes through the communication terminal 90 or a second transmission path that does not pass through the communication terminal 90, The transmitter / receiver 11 transmits the other information determined according to the first transmission path or the second transmission path to the server 50 via the first transmission path or the second transmission path. Here, the server 50 is an example of a destination device, and the communication terminal 90 is an example of an intermediate device.

[0286] This allows optimal data collection and data processing to be performed according to the transmission route when transmitting information from the photographing device 10 to the server 50. In other words, the photographing device 10 can transmit other information determined for each transmission route to the server 50.

[0287] The transmitting / receiving unit 11 may transmit other information simultaneously with the image, or may transmit other information at a timing different from that of transmitting the image.

[0288] [Second aspect] In the first embodiment, the image capturing device 10 includes a data management unit 17 that is an example of a determination unit that determines the other information depending on the first transmission path or the second transmission path. The determination unit may be included in the server 50 or the communication terminal 90. In this case, the transmitting / receiving unit 11 transmits the other information determined by the server 50 or the communication terminal 90 to the server 50 via the first transmission path or the second transmission path.

[0289] [Third aspect] In the first or second mode, the other information is determined according to an application in the communication terminal 90 that processes the reception of the other information from the image capturing apparatus 10 and the transmission of the other information to the server 50.

[0290] This allows other information defined in the application installed in the communication terminal 90 to be transmitted from the photographing device 10 to the server 50 via the communication terminal 90.

[0291] [Fourth aspect] In the first or second aspect, the photographing device 10 is provided with an image processing unit 16 that performs image processing on an image, and the transceiver unit 11 transmits other information to the server 50 via the first transmission path or the second transmission path in association with the image or a processed image on which image processing has been performed, the other information including image identification information indicating whether image processing has been performed or the type of the processed image.

[0292] This allows image identification information indicating whether image processing has been performed or the type of processed image to be transmitted from the photographing device 10 to the server 50, so that the device that performs image processing on the image may be any of the photographing device, communication terminal, and server, realizing flexible image processing. Note that the transmitting / receiving unit 11 does not need to transmit image identification information when the image to be transmitted is a photographed image.

[0293] [Fifth aspect] In the fourth mode, the image processing unit 57 performs a first image processing on the captured image based on image processing setting information, which is information that sets a first image processing to be performed by the photographing device 10 and a second image processing to be performed by the communication terminal 90 or the server 50.

[0294] This allows the image processing to be appropriately divided between the photographing device 10 and the communication terminal 90 or the server 50. Specifically, it is possible to prevent overlapping of the first image processing and the second image processing, or to have the photographing device 10 execute the first image processing related to low-load processing and the communication terminal 90 or the server 50 execute the second image processing related to high-load processing.

[0295] [Sixth aspect] In any of the first to fifth aspects, the other information includes device information of the image capturing device 10. This makes it possible to transmit device information determined for each transmission path from the image capturing device 10 to the server 50.

[0296] [Seventh aspect] A communication terminal 90 according to one embodiment of the present invention includes a transceiver unit 91 as a receiving means for receiving at least one of an image and other information from a photographing device 10 that transmits to a server 50 an image captured by the photographing device 10 and at least one of other information other than an image via a first transmission path that passes through the communication terminal 90, or a second transmission path that does not pass through the communication terminal 90, the other information being information determined according to the first transmission path or the second transmission path, respectively, and the transceiver unit 91 as a transmitting means transmits to the server 50 at least one of the received image, the other information, and second other information different from the other information.

[0297] This makes it possible to transfer images transmitted from the photographing device 10 to the server 50, while also transmitting other information defined for each transmission path from the photographing device 10 to the server 50 via the communication terminal 90, or to transmit second other information in place of the other information to the server 50.

[0298] [Eighth aspect] In the seventh aspect, the communication terminal 90 is provided with an image processing unit 94 that performs image processing on an image received by the transceiver unit 91, and the transceiver unit 91, as a transmitting means, transmits to the server 50 a processed image in which image processing has been performed on the received image, instead of the received image.

[0299] This makes it possible to transfer to the server 50 a processed image obtained by performing image processing on the received image, instead of the received image.

[0300] [Ninth aspect] In the eighth embodiment, the other information includes image identification information indicating whether the image received by the transmission / reception unit 91 is a processed image that has been subjected to image processing by the photographing device 10, or the type of processed image, and the transmission / reception unit 91 receives the image identification information in association with the image, and the image processing unit 94 performs image processing based on the image identification information.

[0301] As a result, for example, if the received image is not a processed image, image processing is performed on the received image, and if the received image is a processed image, image processing is not performed on the received image, thereby ensuring that the processed image is sent to server 50 and preventing the same image processing from being performed twice.

[0302] [Tenth aspect] In the eighth or ninth embodiment, the transmission / reception unit 91 transmits second other information to the server 50 in association with the image received by the transmission / reception unit 91 or a processed image obtained by performing image processing on an image, and the second other information includes second image identification information indicating whether the image to be transmitted to the server 50 is the image received by the transmission / reception unit 91 or the processed image, or the type of the processed image.

[0303] As a result, on the server 50 side, for example, if the received image is not a processed image, image processing is performed on the received image, and if the received image is a processed image, image processing is not performed on the received image, thereby ensuring the acquisition of a processed image and preventing the same image processing from being performed twice.

[0304] It should be noted that, when the received image is a photographed image and the image to be transmitted is also the photographed image, the transmitting / receiving unit 91 does not need to transmit image identification information.

[0305] [Eleventh aspect] In any of the eighth to tenth aspects, the image processing unit 94 performs at least a second image processing on the image received by the transmission / reception unit 91 based on image processing setting information, which is information that sets a first image processing to be performed by the photographing device 10 and a second image processing to be performed by the communication terminal 90.

[0306] This allows the image processing to be appropriately divided between the photographing device 10 and the communication terminal 90. Specifically, it is possible to prevent overlapping of the first image processing and the second image processing, or to have the photographing device 10 execute the first image processing related to low-load processing and the communication terminal 90 execute the second image processing related to high-load processing.

[0307] [12th aspect] In any of the seventh to eleventh aspects, the other information includes device information of the photographing device 10. This makes it possible to transmit device information determined for each transmission path from the photographing device 10 to the server 50 via the communication terminal 90.

[0308] [13th aspect] In any of the seventh to twelfth aspects, the second other information includes management information of the image capturing device 10. This makes it possible to transmit the management information to the server 50.

[0309] [14th aspect] The server 50 according to one embodiment of the present invention includes a transceiver unit 51 that receives at least one of an image captured by the photographing device 10 and other information other than an image from the photographing device 10 via a first transmission path that passes through the communication terminal 90 or a second transmission path that does not pass through the communication terminal 90, and the other information is information determined according to the first transmission path or the second transmission path, respectively.

[0310] This allows optimal data collection and data processing to be performed according to the transmission route when receiving information from the image capturing device 10. In other words, the server 50 can receive other information from the image capturing device 10 that is determined for each transmission route.

[0311] [15th aspect] In the fourteenth aspect, the other information includes image identification information indicating whether the image received by the transmission / reception unit 51 is a processed image obtained by performing image processing on a captured image of a subject, or the type of processed image, and the transmission / reception unit 51 receives the image identification information in association with the image.

[0312] This allows the server 50 to know whether the received image is a processed image or not, and the type of the processed image. If no image identification information is received in association with the image, the received image can be recognized as a captured image.

[0313] [16th aspect] In the fourteenth or fifteenth aspect, the transmitter / receiver 51 receives an image transmitted from the photographing device 10 via the first transmission path, and also receives second other information from the communication terminal 90 that is different from the other information.

[0314] This makes it possible to receive the second other information together with the image from the communication terminal 90.

[0315] [17th aspect] In the sixteenth aspect, the second other information includes second image identification information indicating whether the image received by the transmission / reception unit 51 is a processed image obtained by performing image processing on an image received by the communication terminal 90 from the photographing device 10, or the type of processed image, and the transmission / reception unit 51 receives the second image identification information from the communication terminal 90 in association with the image.

[0316] This allows the server 50 to know whether the received image is a processed image or not, and the type of the processed image. If the second image identification information is not received in association with the image, the received image can be recognized as a captured image.

[0317] [18th aspect] In the fifteenth or seventeenth embodiment, the server 50 includes an image processing unit 57 that performs image processing on the image received by the transmitting / receiving unit based on the image identification information.

[0318] As a result, on the server 50 side, for example, if the received image is not a processed image, image processing is performed on the received image, and if the received image is a processed image, image processing is not performed on the received image, thereby ensuring the acquisition of a processed image and preventing the same image processing from being performed twice.

[0319] [19th aspect] In any of the 14th to 18th aspects, the image processing unit 57 performs at least a second image processing on the image received by the transmission / reception unit 51 based on image processing setting information, which is information that sets a first image processing to be performed by the photographing device 10 or the communication terminal 90 and a second image processing to be performed by the server 50.

[0320] This allows the image processing to be appropriately divided between the photographing device 10 or the communication terminal 90 and the server 50. Specifically, it is possible to prevent overlapping of the first image processing and the second image processing, or to have the photographing device 10 or the communication terminal 90 execute the first image processing related to low-load processing and the server 50 execute the second image processing related to high-load processing.

[0321] [20th aspect] In any of the fourteenth to nineteenth aspects, the other information includes device information of the image capturing device 10. This makes it possible to receive from the image capturing device 10 device information determined for each transmission path.

[0322] [21st aspect] In the sixteenth or seventeenth aspect, the second other information includes management information of the image capturing device 10. This makes it possible to receive the management information from the communication terminal 90.

[0323] [22nd aspect] A transmission method performed by an image capturing device 10 connectable to a server 50 according to one embodiment of the present invention executes a transmission step of transmitting to the server 50 at least one of an image captured by the image capturing device 10 and other information other than the image via a first transmission path passing through a communication terminal 90 or a second transmission path not passing through the communication terminal 90, the other information being information determined according to the first transmission path or the second transmission path, respectively.

[0324] [23rd aspect] A transmission / reception method performed by a communication terminal 90 according to one embodiment of the present invention includes a receiving step of receiving at least one of an image and other information from a photographing device 10 that transmits an image captured by the photographing device 10 and at least one of other information other than the image to a server 50 via a first transmission path that passes through the communication terminal 90 or a second transmission path that does not pass through the communication terminal 90, and a transmitting step of transmitting at least one of the image, the other information, and second other information different from the other information to the server 50, wherein the other information is information determined according to the first transmission path or the second transmission path, respectively.

[0325] [24th aspect] The receiving method executed by the server 50 according to one embodiment of the present invention executes a receiving step of receiving at least one of an image captured by the photographing device 10 and other information other than an image from the photographing device 10 via a first transmission path that passes through the communication terminal 90 or a second transmission path that does not pass through the communication terminal 90, the other information being information determined according to the first transmission path or the second transmission path, respectively.

[0326] [25th aspect] A program executed by an image capturing device 10 connectable to a server 50 according to one embodiment of the present invention causes the computer to execute a transmission step of transmitting to the server 50 at least one of an image captured by the image capturing device 10 and other information other than the image via a first transmission path via a communication terminal 90 or a second transmission path not via the communication terminal 90, the other information being information determined according to the first transmission path or the second transmission path, respectively.

[0327] [26th aspect] A program executed by a communication terminal 90 according to one embodiment of the present invention causes a computer to execute a receiving step of receiving at least one of an image and other information from the photographing device 10, which transmits to the server 50 an image captured by the photographing device 10 and at least one of other information other than the image via a first transmission path that passes through the communication terminal 90 or a second transmission path that does not pass through the communication terminal 90, and a transmitting step of transmitting to the server 50 at least one of the image, the other information, and second other information different from the other information, wherein the other information is information determined according to the first transmission path or the second transmission path, respectively.

[0328] [27th aspect] A program executed by the server 50 according to one embodiment of the present invention causes the computer to execute a receiving step of receiving at least one of an image captured by the photographing device 10 and other information other than an image from the photographing device 10 via a first transmission path that passes through the communication terminal 90 or a second transmission path that does not pass through the communication terminal 90, the other information being information determined according to the first transmission path or the second transmission path, respectively.

[0329] [28th aspect] An image processing system 1, which is an example of an information processing system according to one embodiment of the present invention, includes a photographing device 10 and a server 50. The photographing device 10 includes a transceiver unit 11 that transmits to the server 50 an image captured by the photographing device 10 and / or other information other than images via a first transmission path via a communication terminal 90 or a second transmission path that does not pass through the communication terminal 90. The server 50 includes a transceiver unit 51 that receives images and / or other information from the photographing device 10 via the first transmission path or the second transmission path, and the other information is information determined according to the first transmission path or the second transmission path, respectively.

[0330] [29th aspect] In the 28th embodiment, the image processing system 1 further includes a communication terminal 90, which includes a transceiver unit 91 as a receiving means for receiving images and other information from the photographing device 10, and the transceiver unit 91 as a transmitting means transmits at least one of the image, the other information, and a second other information different from the other information to the server 50.

[0331] ●Additional Information● Each function of the above-described embodiment can be realized by one or more processing circuits. Here, the "processing circuit" in the present embodiment includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), a SOC (System on a chip), a GPU (Graphics Processing Unit), and a conventional circuit module designed to execute each function described above.

[0332] The present invention is not limited to the above-described embodiment, and may be modified within the scope of what a person skilled in the art can conceive, such as adding, changing, or deleting other embodiments, and any aspect is within the scope of the present invention as long as it exhibits the functions and effects of the present invention. [Explanation of symbols]

[0333] 1. Image processing system (an example of an information processing system) 10 Imaging device (an example of an information processing device or a transmitting device) 11 Transmitting / receiving unit (an example of a transmitting means or a receiving means) 12 Operation reception section 13 Shooting control unit (an example of a shooting control means) 14 Videography Department 15 Still Image Shooting Section 16 Image processing section 19 Memory / read unit (an example of a setting means) 50 Server (an example of an information processing device or a destination device) 51 Transmitting / receiving unit (an example of a receiving means or transmitting means) 57 Image Processing Section 59 Memory / read unit (an example of a setting means) 90 Communication terminal (an example of an intermediary device or information processing device) 91 Transmitting / receiving unit (an example of a transmitting means) 92 Reception unit (an example of a setting means) 93 Display control unit (an example of a display control means) 94 Image Processing Section 906 Display (an example of a display unit)

Claims

1. An imaging device connectable to a destination device, a transmission means for transmitting at least one of an image captured by the photographing device and information other than the image to the destination device via a first transmission path that passes through an intermediate device or a second transmission path that does not pass through the intermediate device, The transmitting means is an imaging device that transmits the other information determined depending on the first transmission path or the second transmission path, respectively, to the destination device via the first transmission path or the second transmission path.

2. 2. The photographing apparatus according to claim 1, further comprising a determination unit for determining the other information depending on the first transmission path or the second transmission path.

3. 2 . The photographing device according to claim 1 , wherein the other information is determined in accordance with an application in the intermediary device that processes reception of the other information from the photographing device and transmission of the other information to the destination device.

4. image processing means for executing image processing on the image, the transmitting means transmits the other information to the destination device via the first transmission path or the second transmission path in association with the image or the processed image obtained by executing the image processing; 3. The photographing apparatus according to claim 1, wherein the other information includes image identification information indicating whether the image processing has been executed or the type of the processed image.

5. The photographing device described in claim 4, wherein the image processing means performs the first image processing on the photographed image based on image processing setting information, which is information that sets a first image processing to be performed by the photographing device and a second image processing to be performed by the intermediary device or the destination device.

6. The photographing apparatus according to claim 1 , wherein the other information includes equipment information of the photographing apparatus.

7. a receiving means for receiving, from a photographing device which transmits at least one of an image photographed by the photographing device and information other than the image, the image and the other information via a first transmission path which passes through an intermediate device to a destination device or a second transmission path which does not pass through the intermediate device, the other information is information determined according to each of the first transmission path or the second transmission path, The intermediate device further comprises a transmitting means for transmitting at least one of the image, the other information, and second other information different from the other information to the destination device.

8. an image processing unit that performs image processing on the image received by the receiving unit, 8. The intermediary device according to claim 7, wherein the transmission means transmits to the destination device, in place of the image, a processed image obtained by performing the image processing on the image.

9. the other information includes image identification information indicating whether the image received by the receiving means is a processed image that has been subjected to image processing by the photographing device, or a type of the processed image; the receiving means receives the image identification information in association with the image; 9. The intermediary device according to claim 8, wherein the image processing means executes the image processing based on the image identification information.

10. the transmitting means transmits the second other information to the destination device in association with the image received by the receiving means or a processed image obtained by performing the image processing on the image; The intermediary device according to claim 8 or 9, wherein the second other information includes second image identification information indicating whether the image to be sent to the destination device is the image received by the receiving means or the processed image, or the type of the processed image.

11. The intermediary device of claim 8, wherein the image processing means performs at least the second image processing on the image received by the receiving means based on image processing setting information, which is information that sets a first image processing to be performed by the photographing device and a second image processing to be performed by the intermediary device.

12. The intermediary device according to claim 7 , wherein the other information includes device information of the image capturing device.

13. 8. The intermediary device according to claim 7, wherein the second other information includes management information of the image capturing device.

14. a receiving means for receiving at least one of an image captured by the photographing device and information other than the image from the photographing device via a first transmission path that passes through an intermediary device or a second transmission path that does not pass through the intermediary device; The information processing device, wherein the other information is information determined depending on each of the first transmission path and the second transmission path.

15. the other information includes image identification information indicating whether the image received by the receiving means is a processed image obtained by performing image processing on a photographed image of a subject, or a type of the processed image; 15. The information processing apparatus according to claim 14, wherein the receiving means receives the image identification information in association with the image.

16. 15. The intermediary device according to claim 14, wherein the receiving means receives the image transmitted from the photographing device via the first transmission path, and also receives second other information from the intermediary device that is different from the other information.

17. the second other information includes second image identification information indicating whether the image received by the receiving means is a processed image obtained by performing image processing on the image received by the intermediary device from the photographing device, or a type of the processed image; 17. The information processing apparatus according to claim 16, wherein the receiving means receives the second image identification information from the intermediary apparatus in association with the image.

18. 18. An information processing apparatus according to claim 15, further comprising image processing means for executing image processing on the image received by said receiving means based on the image identification information.

19. The information processing device according to claim 18, wherein the image processing means performs at least the second image processing on the image received by the receiving means based on image processing setting information which is information that sets a first image processing to be performed by the photographing device or the intermediary device and a second image processing to be performed by the information processing device.

20. The information processing apparatus according to claim 14 , wherein the other information includes device information of the photographing device.

21. 18. The information processing apparatus according to claim 16, wherein the second other information includes management information of the image capturing apparatus.

22. A transmission method executed by an imaging device connectable to a destination device, comprising: A transmission step is executed to transmit at least one of an image captured by the photographing device and information other than the image to a destination device via a first transmission path that passes through an intermediate device or a second transmission path that does not pass through the intermediate device; The transmission method, wherein the other information is information determined depending on the first transmission path or the second transmission path.

23. A transmission and reception method executed by an intermediary device, comprising: a receiving step of receiving, from a photographing device that transmits at least one of an image photographed by the photographing device and information other than the image to a destination device via a first transmission path that passes through an intermediate device or a second transmission path that does not pass through the intermediate device, the image and the other information; a transmitting step of transmitting at least one of the image, the other information, and second other information different from the other information to the destination device; The transmission / reception method, wherein the other information is information determined depending on the first transmission path or the second transmission path.

24. A receiving method executed by an information processing device, comprising: executing a receiving step of receiving at least one of an image captured by the photographing device and information other than the image from the photographing device via a first transmission path that passes through an intermediary device or a second transmission path that does not pass through the intermediary device; The receiving method, wherein the other information is information determined depending on the first transmission path or the second transmission path.

25. A program to be executed by an imaging device connectable to a destination device, a transmission step of transmitting at least one of an image captured by the photographing device and information other than the image to a destination device via a first transmission path that passes through an intermediate device or a second transmission path that does not pass through the intermediate device; The other information is information determined depending on each of the first transmission path and the second transmission path.

26. A program to be executed by an intermediary device, a receiving step of receiving, from a photographing device that transmits at least one of an image photographed by the photographing device and information other than the image to a destination device via a first transmission path that passes through an intermediate device or a second transmission path that does not pass through the intermediate device, the image and the other information; a transmitting step of transmitting at least one of the other information and second other information different from the other information to the destination device; The other information is information determined depending on each of the first transmission path and the second transmission path.

27. A program to be executed by an information processing device, a receiving step of receiving at least one of an image captured by the photographing device and information other than the image from the photographing device via a first transmission path that passes through an intermediary device or a second transmission path that does not pass through the intermediary device; The other information is information determined depending on each of the first transmission path and the second transmission path.

28. An information processing system including an imaging device and the transmission destination device, The imaging device is a transmission means for transmitting at least one of an image captured by the photographing device and information other than the image to the destination device via a first transmission path that passes through an intermediate device or a second transmission path that does not pass through the intermediate device, The destination device is a receiving means for receiving at least one of the image and the other information from the photographing device via the first transmission path or the second transmission path, The other information is information determined depending on each of the first transmission path and the second transmission path.

29. The intermediary device further comprises: The intermediary device is a receiving means for receiving the image and the other information from the photographing device; 29. The information processing system according to claim 28, further comprising a transmitting means for transmitting at least one of the image, the other information, and second other information different from the other information to the destination device.

Images