Imaging device, mediation device, information processing device, transmission method, and reception method

Abstract

The invention relates to an imaging device, a mediation device, an information processing device, a transmission method, a transmission / reception method, a reception method, a computer program product, and an information processing system. The present invention executes the most suitable data collection and data processing according to a transmission path when transmitting information from an imaging device to a transmission destination device. An imaging device (10) is provided with a transmission / reception unit (11) that transmits, to a server (50), an image captured by the imaging device (10) and / or other information other than the image via a first transmission path via a communication terminal (90) or a second transmission path not via the communication terminal (90). The transmission / reception unit (11) transmits, to the server (50) via the first transmission path or the second transmission path, other information determined on the basis of the first transmission path or the second transmission path.

Description

Technical Field

[0001] The present invention relates to a photographing device, an intermediary device, an information processing device, a sending method, a sending and receiving method, a receiving method, a computer program product and an information processing system. Background Art

[0002] In edge devices such as cameras, there are edge devices that can not only upload device information and data (files, images, etc.) stored in the device to the cloud server indirectly (via intermediary devices such as smartphones and PCs), but also upload them to the cloud server directly (without intermediary devices), distinguishing between edge devices using multiple upload paths.

[0003] Patent Document 1 describes a distributed system in which edge devices collaborate with computers such as cloud servers, provide a technology for processing the internal state of a controller in the edge devices, and achieve universalization of data communications used for the collaboration and reduction of data communication volume.

[0004] However, in Patent Document 1, although data can be collected using a plurality of paths or networks, it does not take into account optimal data collection and data processing based on the plurality of paths.

[0005] Patent Document 1: Japanese Patent Application Publication No. 2022-081952 Summary of the invention

[0006] An object of the present invention is to perform optimal data collection and data processing according to a transmission path when information is transmitted from a photographing device to a transmission destination device.

[0007] In order to achieve the above-mentioned purpose, the photographic device involved in technical solution 1 of the present invention can be connected to a destination device, and includes a sending mechanism, which sends at least one of an image taken by the photographic device and other information other than the image to the destination device through a first sending path via an intermediary device, or a second sending path not via the intermediary device; the sending mechanism sends the other information determined according to the first sending path or the second sending path to the destination device through the first sending path or the second sending path.

[0008] The effect of the present invention is described below:

[0009] According to the present invention, when information is transmitted from a photographing device to a transmission destination device, optimal data collection and data processing can be performed according to the transmission path. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a diagram showing an example of the overall configuration of an image processing system according to an embodiment of the present invention.

[0011] Figure 2 This is a diagram showing an example of a full-sphere image captured by a photographic device.

[0012] Figure 3 (A) is a diagram showing a hemispherical image (front) taken by a photographic device, (B) is a diagram showing a hemispherical image (back) taken by a photographic device, and (C) is a diagram showing an image represented by an equirectangular graph method.

[0013] Figure 4 (A) is a conceptual diagram showing a state where the sphere is covered with an equirectangular projection image, and (B) is a diagram showing an image of the entire celestial sphere.

[0014] Figure 5 This is a diagram showing the positions of a virtual camera and a predetermined area when the entire celestial sphere image is formed into a three-dimensional stereoscopic sphere.

[0015] Figure 6 This is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T.

[0016] Figure 7 This is a diagram showing an example of a state when photographing by a photographing device.

[0017] Figure 8 (A) to (B) are diagrams for explaining an example of a full celestial sphere image.

[0018] Fig. 9 (A) to (B) are diagrams for explaining an example of a planar image converted from a full celestial sphere image.

[0019] Fig.10 (A) to (B) are diagrams showing an example of an imaging device applicable to an image processing system.

[0020] Fig.11 This is a diagram showing an example of the hardware configuration of the imaging device according to this embodiment.

[0021] Fig.12 This is a diagram showing an example of the hardware configuration of the server and communication terminal involved in this embodiment.

[0022] Fig.13 It is a diagram showing an example of the functional configuration of the image processing system according to the present embodiment.

[0023] Fig.14 (A) to (B) are diagrams showing another example of the functional configuration of the communication terminal according to the present embodiment.

[0024] Fig.15 (A) to (C) are conceptual diagrams showing an example of the account management table involved in this embodiment.

[0025] Fig.16 (A) to (C) are conceptual diagrams showing an example of a setting information management table according to the present embodiment.

[0026] Fig.17 It is a conceptual diagram showing an example of a device information management table involved in this embodiment.

[0027] Fig.18 This is a sequence diagram showing an example of registration processing by direct transmission by the server according to the present embodiment.

[0028] Fig.19 (A) to (B) are sequence diagrams respectively showing an example of server direct transmission of an image and an example of server direct transmission of device information involved in this embodiment.

[0029] Fig. 20 This is a sequence diagram showing an example of the registration process of the sending and receiving agent involved in this embodiment.

[0030] Fig.21 This is a sequence diagram showing an example of the image sending and receiving intermediary processing involved in this embodiment.

[0031] Fig. 22 This is a sequence diagram showing an example of the intermediary processing of sending and receiving device information involved in this embodiment.

[0032] Fig.23 This is a sequence diagram showing an example of registration processing of another sending and receiving agent involved in this embodiment.

[0033] Fig.24 This is a sequence diagram showing an example of another image sending and receiving intermediary processing involved in this embodiment.

[0034] Fig.25 This is a sequence diagram showing an example of another sending and receiving intermediary processing of device information involved in this embodiment.

[0035] Fig.26 This is a sequence diagram showing an example of the display process of the device information involved in this embodiment.

[0036] Fig. 27 This is a sequence diagram showing another example of the display process of the device information involved in this embodiment.

[0037] Fig.28 This is a flowchart showing an example of image transmission processing in the imaging device involved in this embodiment.

[0038] Fig.29(A) to (B) are flowcharts showing an example of image transmission and reception processing in the communication terminal and image reception processing in the server according to the present embodiment.

[0039] Fig.30 This is a flowchart showing an example of a process of sending device information in the photographing apparatus according to this embodiment.

[0040] Fig.31 (A) to (B) are diagrams showing examples of display screens of device information according to the present embodiment.

[0041] Fig.32 It is a diagram showing an example of a display screen of management information according to the present embodiment.

[0042] Fig.33 It is a diagram showing another example of the display screen of the management information involved in this embodiment.

[0043] Fig.34 It is a diagram showing an example of the overall configuration of an image processing system according to a modification example of the embodiment of the present invention.

[0044] Fig.35 It is a diagram showing an example of the overall configuration of an image processing system according to a second modified example according to the embodiment of the present invention.

[0045] The symbols in the figure are explained as follows:

[0046] 1 Image processing system (an example of an information processing system)

[0047] 10 Photographic device (an example of an information processing device or a transmission destination device)

[0048] 11 Transmitting and receiving unit (an example of a transmitting mechanism or a receiving mechanism)

[0049] 12 Operation Reception Department

[0050] 13 Photography control unit (an example of a photography control mechanism)

[0051] 14Moving Image Photography Department

[0052] 15Still Image Photography Department

[0053] 16Image Processing Unit

[0054] 19 Storage / reading unit (an example of setting mechanism)

[0055] 50 Server (an example of an information processing device, a transmission destination device)

[0056] 51 Transmitting and receiving unit (an example of a transmitting mechanism or a receiving mechanism)

[0057] 57 Image Processing Unit

[0058] 59 Storage / reading unit (an example of setting mechanism)

[0059] 90 Communication terminal (an example of an intermediary device or information processing device)

[0060] 91 Transmission and reception unit (an example of a transmission mechanism)

[0061] 92 Receiving Department (an example of an establishment organization)

[0062] 93 Display control unit (an example of a display control mechanism)

[0063] 94 Image Processing Unit

[0064] 906 display (an example of a display unit) DETAILED DESCRIPTION

[0065] Hereinafter, the mode for implementing the invention will be described with reference to the drawings. In the description of the drawings, the same reference numerals are given to the same elements, and repeated descriptions are omitted.

[0066] ●Overview of image processing system

[0067] Figure 1 It is a diagram showing an example of the overall configuration of an image processing system according to an embodiment of the present invention. Figure 1 The image processing system 1 shown is an example of an information processing system, and is a system that performs image processing on photographic images for allowing viewers to browse online the interior space of a structure such as a real estate or a building, or a structure such as a construction or civil engineering.

[0068] like Figure 1 As shown, the image processing system 1 includes a photographic device 10, a server 50 and a communication terminal 90. The server 50 and the communication terminal 90 constituting the image processing system 1 can communicate with each other via a communication network 100. The photographic device 10 can communicate with the communication terminal 90 through short-range wireless communication such as Wi-Fi, but can also communicate with the server 50 and the communication terminal 90 through the communication network 100. The communication network 100 is composed of the Internet, a mobile communication network, a LAN (local area network), etc. In addition, the communication network 100 can include not only wired communication, but also networks based on wireless communication such as 3G (third generation), 4G (fourth generation), 5G (fifth generation), Wi-Fi (wireless Internet access) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access) or LTE (Long Term Evolution).

[0069] The server 50 is an example of an information processing device or a destination device, and is a server computer that performs image processing on the internal space of a structure such as a real estate or a building, or a photographic image of a structure such as a building or a civil engineering structure, which is a predetermined location. For example, the server 50 obtains the photographic image taken by the photographic device 10, and uses the obtained photographic image to generate a travel image for providing a virtual tour to the user. Here, the so-called virtual tour is a content that allows the user to browse as if browsing real estate items, etc. on the spot. In addition, the travel image is a browsing image that is generated using a plurality of photographic images taken by the photographic device 10 and can be virtually moved in the location captured in the photographic image by user operation. Such a virtual tour is not limited to the internal space of a structure such as a real estate or a construction site, and can also be appropriately realized in the external space of a structure such as a tourist attraction or a theme park. That is, the internal space of the structure can also be captured as a predetermined area. In the case of realizing a virtual tour in the external space of the structure, the present embodiment can be realized by replacing the map representing the internal space of a structure such as a real estate or a building, which will be described later, with, for example, a tourist attraction map or an area map of a theme park that introduces tourist attractions.

[0070] In addition, the server 50 may be composed of one server computer or a plurality of server computers. In addition, the server 50 is described as a server computer existing in a cloud environment, but it may also be a server existing in a local environment.

[0071] The photographing device 10 is an example of an information processing device and a transmission source device, and is a special digital camera (full celestial spherical photographing device) that can omnidirectionally photograph the space where a structure such as real estate exists at the photographing location and obtain a full celestial sphere (360° in both the circumferential direction and the vertical direction). The following description mainly takes structures such as real estate as an example, but as mentioned above, it may also be an internal space of a structure such as a building, a construction, a civil engineering structure, etc.

[0072] In addition, instead of the photographing device 10, for example, a communication terminal 90 having a function of acquiring a full celestial sphere image may be used to acquire a full celestial sphere image, or a camera accessory capable of acquiring a full celestial sphere image may be connected to the communication terminal 90 to acquire a full celestial sphere image through the communication terminal 90 and the camera accessory. That is, the so-called transmission source device may be any device as long as it transmits the acquired full celestial sphere image to the server 50.

[0073] In addition, the so-called spherical image refers to an image of a so-called solid angle of 4πsr [sr: steradian]. In addition, in this specification, for convenience, an image lacking a part of the spherical image is also referred to as a spherical image. For example, it is an image lacking a part directly above or directly below the spherical photography device, a part in the vertical upward direction or vertical downward direction of the spherical image, or a part of a specified area of ​​the spherical image.

[0074] This can be used in scenarios where, for example, when viewing a full spherical image, the user does not pay much attention to the portion directly above or directly below the subject in the full spherical image. In this case, the imaging element and the optical system are designed so that the portion is not captured at all, the image is not displayed, a logo is displayed on the portion, etc., and it is also conceivable that the full spherical image itself is not displayed.

[0075] The photographic device 10 is used, for example, by real estate practitioners who manage or sell real estate. In addition, the photographic device 10 may be a wide-angle camera or a stereo camera that can obtain wide-angle images with a viewing angle greater than a specified value. A wide-angle image is generally an image taken using a wide-angle lens (wide-angle image), and is an image taken using a lens that can capture a wider range than the human eye can perceive. In other words, the photographic device 10 is a photographic mechanism that can obtain images (full spherical images, wide-angle images) taken using a lens with a focal length shorter than a specified value. A wide-angle image generally refers to an image taken with a lens with a focal length of less than 35 mm converted to 35 mm film. In addition, the photographic device 10 may have a panoramic photography function in its photography function, and may also capture panoramic images.

[0076] Here, when sending an image from the photographing device 10 to the server 50 , the image may not be sent directly from the photographing device 10 to the server 50 due to the communication environment, etc. In this case, the image is sent from the photographing device 10 to the server 50 via the communication terminal 90 .

[0077] The communication terminal 90 is an example of an intermediary device or an information processing device, and is a computer such as a tablet terminal. The communication terminal 90 is used, for example, by the same real estate developer as the photographic device 10. The communication terminal 90 may be installed with, for example, a dedicated application for giving shooting instructions to the photographic device 10 and for viewing images provided by the server 50, and an application related to sending and receiving that mediates the sending and receiving of information between the photographic device 10 and the server 50. In addition, the communication terminal 90 may be configured to give shooting instructions and view images by accessing a dedicated Web site using a Web browser instead of using a dedicated application. In addition, shooting instructions and image viewing may be performed by different communication terminals 90. In the following, application programs are simply recorded as applications.

[0078] The communication terminal 90 is not limited to a tablet terminal, and may be, for example, a PC, a smartphone, a wearable terminal, an HMD (head mounted display), or an IWB (interactive whiteboard: a whiteboard having an electronic blackboard function capable of mutual communication).

[0079] In the image processing system 1 described above, the photographic device 10 and the server 50 have an API, and the communication terminal 90 has an application that mediates the sending and receiving of information between the photographic device 10 and the server 50, thereby enabling images to be sent from the photographic device 10 to the server 50 via the communication terminal 90. However, when sending device information of the photographic device 10 such as the battery remaining level or the storage remaining level to the server 50, there may be restrictions.

[0080] For example, when the type, data format, communication protocol, etc. of device information sent from the photographic device 10 to the server 50 is changed, due to the time delay in the development of functional updates of the photographic device 10 and functional updates of the applications of the communication terminal 90, for example, the types of device information sent from the photographic device 10 to the server 50 via the communication terminal 90 may be reduced compared to the types of device information sent directly from the photographic device 10 to the server 50.

[0081] This problem is particularly prominent when using an application developed by a company different from the manufacturer of the API. Compared with the types of device information sent from the photographic device 10 to the server 50 via the communication terminal 90 equipped with the application developed by a company different from the manufacturer of the API, the types of device information sent from the photographic device 10 to the server 50 via the communication terminal 90 equipped with the application developed by the company different from the manufacturer of the API may be reduced.

[0082] In the above situation, if the photographing apparatus 10 uniformly transmits various types of device information regardless of the type of transmission path when directly transmitting to the server 50 , a reception error may occur when transmitting via the communication terminal 90 .

[0083] On the other hand, if the photographic device 10 sends a smaller number of types of device information regardless of the type of transmission path via a communication terminal 90 equipped with an application developed by another company, when sending directly to the server 50, it is restricted from receiving device information that it could have received.

[0084] In addition, the above-mentioned subject is not limited to the device information of the photographing device 10, but also applies to the image recognition information. That is, the image recognition information indicates whether the image sent by the photographing device 10 has been subjected to image processing or the type of image processing performed, but depending on the corresponding status of the application, it may not be possible to send and receive the image recognition information via the communication terminal 90.

[0085] In this situation, if the photographing device 10 uniformly transmits the image identification information regardless of the type of transmission path when directly transmitting to the server 50 , a reception error may occur when transmitting via the communication terminal 90 .

[0086] On the other hand, if the photographic device 10 does not send image recognition information regardless of the type of transmission path via the communication terminal 90 equipped with an application developed by another company that is the slowest in development, the server 50 cannot determine whether the image is to be processed or the type of the processed image, so no image processing is performed on the photographic device 10 side, and all image processing is restricted to be performed on the server 50 side. The above-mentioned issues are not limited to the case where the photographic device 10 and the server 50 have an API, but also apply to the case where the communication terminal 90 has an API.

[0087] The purpose of this embodiment is to perform optimal data collection and data processing according to the transmission path when device information, image recognition information, and other information are transmitted from the imaging device to the transmission destination device as described above.

[0088] ○Outline of the Photographic Equipment○

[0089] Here, use Figures 2 to 11 , the overview of the photographing device 10 constituting the image processing system 1 is described. Figure 2 This is a diagram showing an example of a full-sphere image captured by a photographic device. Figure 2 The image shown is a full spherical image of a room of a real estate as an example of the internal space of a structure, which is taken by the photographing device 10. Since the full spherical image can take pictures of the interior of the room in all directions, it is suitable for viewing real estate, etc. The full spherical image has various forms, but most of them are generated by the projection method of equirectangular (equirectangular projection method) described later. The advantage of the image generated by the equirectangular projection method is that the image shape is rectangular, the image data is saved efficiently and easily, there is little deformation near the equator, and the vertical straight lines are not deformed, so it looks more natural.

[0090] ○ Method for generating full celestial sphere images

[0091] Next, use Figures 3 to 9 The following describes how to generate a full celestial sphere image. First, use Figure 3 and Figure 4 , an overview of the processing from an image captured by the photographing device 10 to the generation of a full celestial sphere image is described. Figure 3 (A) is a diagram showing a hemispherical image (front side) taken by a photographic device, Figure 3 (B) is a diagram showing a hemispherical image (back side) taken by a photographic device. Figure 3 (C) is a diagram showing an image represented by the equirectangular projection method (hereinafter referred to as an "equirectangular projection image"). Figure 4 (A) is a conceptual diagram showing a state where a sphere is covered with an equirectangular projection image. Figure 4 (B) is a diagram showing a full celestial sphere image.

[0092] The photographing device 10 is provided with imaging elements on the front side (front side) and the back side (rear side). These imaging elements (image sensors) are used together with optical components such as lenses that can capture hemispherical images (viewing angle of 180° or more). The photographing device 10 can obtain two hemispherical images by using two imaging elements to capture the subject around the user respectively.

[0093] like Figure 3 (A) and Figure 3 As shown in (B), the image obtained by the imaging element of the imaging device 10 is a curved hemispherical image (front side and rear side). Then, the imaging device 10 synthesizes the hemispherical image (front side) and the hemispherical image (rear side) inverted by 180 degrees to produce a Figure 3 (C) shows the equirectangular projection image EC.

[0094] Then, the photographing device 10 uses OpenGLES (Open Graphics Library for Embedded Systems), such as Figure 4 As shown in (A), the equidistant cylindrical projection image EC is pasted in a manner of covering the spherical surface to produce Figure 4 The whole celestial sphere image (whole celestial sphere panoramic image) CE shown in (B). In this way, the whole celestial sphere image CE is represented as an image of the equirectangular projection image EC toward the center of the sphere. In addition, OpenGLES is a graphics library for visualizing 2D (2-Dimensions) and 3D (3-Dimensions) data. In addition, the whole celestial sphere image CE can be a static image or a dynamic image. Furthermore, the transformation method is not limited to OpenGLES, as long as it is a method that can transform from a hemispherical image to an equirectangular projection method, for example, it can also be a CPU-based operation or an OpenCL-based operation.

[0095] As described above, since the celestial sphere image CE is an image pasted in a manner that covers the spherical surface, it may look unnatural to humans. Therefore, the photographing device 10 can display a predetermined area T (hereinafter referred to as a "predetermined area image") of a part of the celestial sphere image CE as a flat image with less curvature, thereby making it possible to display the image without giving people a sense of unnaturalness. Figure 5 and Figure 6 Provide explanation.

[0096] Figure 51 is a diagram showing the positions of a virtual camera and a predetermined area when the entire celestial sphere image is set as a three-dimensional stereoscopic sphere. The virtual camera IC corresponds to the position of the viewpoint of the user viewing the entire celestial sphere image CE displayed as a three-dimensional stereoscopic sphere. Figure 5 The celestial sphere image CE is represented by a three-dimensional stereoscopic sphere CS. When the celestial sphere image CE generated in this way is a stereoscopic sphere CS, Figure 5 As shown, the virtual camera IC is located inside the celestial sphere image CE. The specified area T in the celestial sphere image CE is the photographic area of ​​the virtual camera IC, and is determined by the specified area information representing the photographic direction and viewing angle of the virtual camera IC in the three-dimensional virtual space containing the celestial sphere image CE. In addition, the zoom of the specified area T can also be expressed by making the virtual camera IC approach or move away from the celestial sphere image CE. The specified area image Q is an image of the specified area T in the celestial sphere image CE. Therefore, the specified area T can be determined by the viewing angle α and the distance f from the virtual camera IC to the celestial sphere image CE.

[0097] Then, the prescribed area image Q is displayed on a prescribed display as an image of the photographing area of ​​the virtual camera IC. The following description uses the photographing direction (ea, aa) and the angle of view (α) of the virtual camera IC. In addition, the prescribed area T may be represented by the photographing area (X, Y, Z) of the virtual camera IC as the prescribed area T instead of the angle of view α and the distance f.

[0098] Next, use Figure 6 The relationship between the predetermined region information and the image of the predetermined region T will be described. Figure 6 is a diagram showing the relationship between the specified area information and the image of the specified area T. Figure 6 As shown,

[0099] "ea" represents the elevation angle, "aa" represents the azimuth angle, and "α" represents the viewing angle. That is, the posture of the virtual camera IC is changed so that the focus point of the virtual camera IC represented by the photographing direction (ea, aa) becomes the center point CP (x, y) of the prescribed area T which is the photographing area of ​​the virtual camera IC. Figure 6 As shown in FIG. 1 , it is assumed that the center point CP (x, y) of the specified area T represented by the angle of view α of the virtual camera IC is the parameter (x, y) of the specified area information when the diagonal angle of view α is α. The specified area image Q is the image of the specified area T in the celestial sphere image CE. f is the distance from the virtual camera IC to the center point CP (x, y). L is the distance between any vertex of the specified area T and the center point CP (x, y) (2L is the diagonal line). Then, Figure 6 In the equation (1), the following trigonometric function usually holds true.

[0100] [Formula 1]

[0101] L / f = tan(α / 2)… (Formula 1)

[0102] Next, use Figure 7 A description will be given of a situation in which photography is performed by the photography device 10. Figure 7 1 is a diagram showing an example of a state when a photographing device is used to take a picture. In order to be able to take a picture while looking around the entire room of a real estate, etc., it is preferable to set the photographing device 10 at a position close to the height of human eyes. Figure 7 As shown, the photographic device 10 is generally fixed with a support member 20 such as a tripod or a tripod to shoot. As described above, the photographic device 10 is a celestial spherical photographic device that can obtain light from all directions around it, or it can be said that it obtains an image on a unit sphere around the photographic device 10 (celestial spherical image CE). If the shooting direction is determined, the photographic device 10 determines the coordinates of the celestial spherical image. For example, in Figure 7 In the figure, point A is located at a distance (d, -h) from the center point C of the photographic device 10. At this time, if the angle formed with the line segment AC in the horizontal direction is set to θ, the angle θ can be expressed by the following (Formula 2).

[0103] [Formula 2]

[0104] θ = arctan(h / d)… (Formula 2)

[0105] Furthermore, assuming that point A is at a depression angle θ, the distance d between point A and point B can be expressed by the following (Formula 3) using the installation height h of the imaging device 10 .

[0106] [Formula 3]

[0107] d = h / tanθ (Formula 3)

[0108] Here, a process of converting position information on the entire spherical image from the entire spherical image into coordinates on the converted planar image will be described in general terms. Figure 8 is a diagram for explaining an example of a full celestial sphere image. Figure 8 (A) is connected by wire Figure 3 (A) shows a hemispherical image with the same incident angles in the horizontal and vertical directions relative to the optical axis. Hereinafter, the incident angle in the horizontal direction relative to the optical axis is referred to as "θ", and the incident angle in the vertical direction relative to the optical axis is referred to as "φ".

[0109] in addition, Fig. 9 (A) is a diagram illustrating an example of an image processed by the isometric column method. Figure 8The image shown corresponds to the pre-generated LUT (lookup table), etc., and is processed by the isometric column method. Figure 8 (A) and Figure 8 After the images shown in (B) are synthesized, the photographing device 10 generates a corresponding full celestial sphere image. Fig. 9 The planar image shown in (A). Figure 3 The equirectangular projection image EC shown in (C) is Fig. 9 An example of a planar image shown in (A).

[0110] like Fig. 9 As shown in (A), in the image processed by the equirectangular method, longitude (θ) and latitude (φ) are orthogonal. Fig. 9 In the example shown in (A), by indicating the center of the image as (0, 0), the latitude direction as -90 to +90, and the longitude direction as -180 to +180, any position in the celestial sphere image can be indicated. For example, the coordinates of the upper left corner of the image are (-180, -90). In addition, the coordinates of the celestial sphere image can be expressed as follows: Fig. 9 The coordinates of the celestial sphere image can be transformed into the following: Fig. 9 It is represented by the two-dimensional coordinates (x, y) shown in (B).

[0111] in addition, Fig. 9 (A) or Fig. 9 The synthesis process of the planar image shown in (B) is not limited to simply arranging the Figure 8 (A) and Figure 8 For example, when the horizontal center of the full celestial sphere image is not θ=180°, in the synthesis process, the photographing device 10 first Figure 3 The hemispherical image shown in (C) is preprocessed and arranged at the center of the full celestial image. Next, the photographing device 10 may also place the corresponding Figure 3 The hemispherical image shown in (B) is preprocessed and divided into sizes that can be configured in the left and right parts, and the hemispherical image is synthesized to generate Figure 3 (C) shows the equirectangular projection image EC.

[0112] In addition, Fig. 9 In the plane image shown in (A), Figure 8 (A) and Figure 8 The position corresponding to the pole (PL1 or PL2) of the hemispherical image (global celestial image) shown in (B) is the line segment CT1 or CT2. Figure 4 (A) and Figure 4 As shown in (B), the full celestial sphere image (for example, the full celestial sphere image CE) is obtained by using OpenGLES. Fig. 9 The plane image (equirectangular projection image EC) shown in (A) is pasted on a spherical surface.

[0113] ○ Examples of photography devices that can be applied to image processing systems

[0114] Fig.10 This is a schematic diagram for explaining an example of a photographic device that can be applied to an image processing system. Fig.10 (A) represents a special photographic device having multiple imaging elements capable of generating a full celestial spherical image by the above-described generation method. The special photographic device uses an optical system composed of a wide-angle lens or a fisheye lens with a wide viewing angle, and can combine the outputs of multiple imaging elements to obtain an all-round image. Fig.10 (B) represents a general photographic device as a so-called ordinary camera. A general photographic device is, for example, a common digital camera or a portable terminal such as a smart phone equipped with a camera. The photographer holds the general photographic device and rotates it while shooting. The general photographic device can obtain an all-round image by synthesizing the obtained image. Both the special photographic device and the general photographic device connect multiple photographic results through image processing (stitching processing) to generate a final photographic image. In addition, the optical centers of the photographic devices 10 used to obtain multiple photographic results are preferably the same.

[0115] ●Hardware composition

[0116] Next, use Fig.11 and Fig.12 The hardware structure of each device or terminal constituting the image processing system involved in the embodiment is described. Fig.11 and Fig.12 The hardware configuration shown may also include or delete components as needed.

[0117] ○Hardware structure of the photographic device○

[0118] First, use Fig.11 The hardware structure of the imaging device 10 will be described. Fig.11 1 is a diagram showing an example of the hardware structure of a photographic device. Hereinafter, the photographic device 10 is a full-sphere (omnidirectional) photographic device using two imaging elements, but the number of imaging elements may be two or more. In addition, it is not necessarily a device dedicated to omnidirectional photography, and a rear omnidirectional photographic mechanism may be installed on a common digital camera or smart phone, thereby having substantially the same function as the photographic device 10. .

[0119] like Fig.11As shown, the photographing device 10 includes a camera unit 101, an image processing unit 104, a camera control unit 105, a microphone 108, a sound processing unit 109, a CPU (central processing unit)

[0120] 111, ROM (read only memory) 112, SRAM (static random access memory) 113, DRAM (dynamic random access memory) 114, operation unit 115, input / output I / F (interface) 116, short-distance communication circuit 117, antenna 117a of short-distance communication circuit 117, electronic compass 118, gyro sensor 119, acceleration sensor 120, and network I / F 121. In addition, there is an ISP (image signal processor) as a processor for processing image signals.

[0121] The imaging unit 101 includes optical systems (wide-angle lenses or so-called fisheye lenses) 102a and 102b (hereinafter referred to as lenses 102 when no distinction is required) with a viewing angle of 180° or more for capturing hemispherical images, and two imaging elements 103a and 103b provided corresponding to the lenses. Here, a combination of an optical system and an imaging element is referred to as a photographic optical system, and the photographic device 10 can be formed by arranging the two photographic optical systems to face each other. In addition, the photographic device 10 can also be formed by using more than two photographic optical systems. The imaging elements 103a and 103b include image sensors such as CMOS (complementary metal oxide semiconductor) sensors and CCD (charge coupled device) sensors that convert optical images generated by the lenses 102a and 102b into image data of electrical signals and output them, a timing generation circuit that generates horizontal or vertical synchronization signals or pixel clocks of the image sensors, and a register group that sets various commands or parameters required for the operation of the imaging elements.

[0122] The imaging elements 103a and 103b of the imaging unit 101 are connected to the image processing unit 104 via a parallel I / F bus, respectively. On the other hand, the imaging elements 103a and 103b of the imaging unit 101 are connected to the imaging control unit 105 via a serial I / F bus (I2C bus, etc.), respectively. The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. In addition, the ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the input / output I / F 116, the short-distance communication circuit 117, the electronic compass 118, the gyro sensor 119, the acceleration sensor 120, the network I / F 121, etc. are also connected to the bus 110.

[0123] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via the parallel I / F bus, performs a predetermined process on each image data, and then performs a synthesis process on the image data to generate a Figure 3 (C) shows the data of the positive cylindrical projection image.

[0124] The camera control unit 105 generally uses the camera control unit 105 as a master device and the camera elements 103a and 103b as slave devices, and uses the I2C bus to set commands to the register groups of the camera elements 103a and 103b. It receives necessary commands from the CPU 111. In addition, the camera control unit 105 also uses the I2C bus to read the status data of the register groups of the camera elements 103a and 103b and send it to the CPU 111.

[0125] In addition, the image capture control unit 105 instructs the image capture elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Depending on the imaging device 10, there may be a function corresponding to a preview display function or a dynamic image display function on a display (for example, a display of an external terminal such as a smartphone that performs short-range communication with the imaging device 10 using the short-range communication circuit 117). In this case, the output of image data from the image capture elements 103a and 103b is performed continuously at a predetermined frame rate (frames / second).

[0126] In addition, as described later, the image capture control unit 105 also functions as a synchronization control mechanism that cooperates with the CPU 111 to synchronize the output timing of the image data of the image capture elements 103a and 103b. In addition, in the present embodiment, the image capture device 10 is not provided with a display unit (display), but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 obtains the sound data output from the microphone 108 through the I / F bus and performs a predetermined process on the sound data.

[0127] The CPU 111 controls the overall operation of the imaging device 10 and performs necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are working memories that store programs executed by the CPU 111 and data being processed. In particular, the DRAM 114 stores image data being processed by the image processing unit 104 and data of processed equirectangular projection images.

[0128] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, etc. The user inputs various shooting modes, shooting conditions, etc. by operating the operation unit 115 .

[0129] The input / output I / F 116 is a general term for an interface circuit (SD memory card I / F or USB I / F, etc.) with an external medium such as an SD memory card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded in an external medium via the input / output I / F 116, or is sent to an external terminal (device) via the input / output I / F 116 as needed.

[0130] The short-range communication circuit 117 communicates with an external terminal (device) through a short-range wireless communication technology such as NFC (near field communication), Bluetooth (registered trademark) or Wi-Fi via an antenna 117a provided in the photographing device 10. The short-range communication circuit 117 can send data of an equirectangular projection image to an external terminal (device).

[0131] The electronic compass 118 calculates the orientation of the photographic device 10 based on the magnetic force of the earth and outputs orientation information. This orientation information is an example of associated information (metadata) along Exif, and is used for image processing such as image correction of photographic images. In addition, the associated information also includes data such as the shooting date and time of the image and the data capacity of the image data. In addition, the gyro sensor 119 is a sensor that detects the change in angle (roll angle, pitch angle, yaw angle) caused by the movement of the photographic device 10. The change in angle is an example of associated information (metadata) along Exif, and is used for image processing such as image correction of photographic images. Furthermore, the acceleration sensor 120 is a sensor that detects acceleration in three-axis directions. The photographic device 10 calculates the posture (angle relative to the direction of gravity) of the device (photographic device 10) based on the acceleration detected by the acceleration sensor 120. The photographic device 10 improves the accuracy of image correction by providing the acceleration sensor 120. The network I / F 121 is an interface for data communication using the communication network 100 such as the Internet via a router or the like.

[0132] ○Server hardware configuration○

[0133] First, use Fig.12 The hardware structure of the server 50 will be described. Fig.12 50 is a diagram showing an example of a hardware structure of a server. Each hardware structure of the server 50 is represented by a 500 series symbol. The server 50 is constructed by a computer, such as Fig.12As shown, it includes a CPU 501, a ROM 502, a RAM (random access memory) 503, an HD (hard disk) 504, an HDD (hard disk drive) controller 505, a display 506, an external device connection I / F 508, a network I / F 509, a bus 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk-Rewritable) drive 514 and a media I / F 516.

[0134] Among them, CPU501 controls the overall operation of server 50. ROM502 stores programs for driving CPU501 such as IPL (boot program). RAM 503 is used as a work area for CPU 501. HD504 stores various data such as programs. HDD controller 505 controls the reading or writing of various data to HD504 under the control of CPU501. Display 506 displays various information such as cursor, menu, window, text or image. In addition, display 506 can also be a touch panel display with an input mechanism. External device connection I / F508 is an interface for connecting various external devices. The external device in this case is, for example, a USB memory, etc. Network I / F 509 is an interface for data communication using communication network 100. Bus 510 is for electrical connection Fig.12 The address bus and data bus of each component such as the CPU 501 shown.

[0135] In addition, the keyboard 511 is a type of input mechanism having a plurality of keys for inputting text, numerical values, various instructions, etc. The pointing device 512 is a type of input mechanism for selecting or executing various instructions, selecting a processing object, or moving a cursor, etc. In addition, the input mechanism can be not only the keyboard 511 and the pointing device 512, but also a touch panel or a sound input device, etc. The DVD-RW drive 514 controls the reading or writing of various data of the DVD-RW 513 as an example of a removable recording medium. In addition, the removable recording medium is not limited to DVD-RW, but can also be a DVD-R or Blu-ray (registered trademark) Disc, etc. The medium I / F 516 controls the reading or writing (storage) of data of the recording medium 515 such as a flash memory.

[0136] ○Hardware structure of communication terminal○

[0137] Fig.12 The figure also shows an example of the hardware structure of the communication terminal 90. Each hardware structure of the communication terminal 90 is represented by a symbol in the 900 series in parentheses. The communication terminal 90 is constructed by a computer, such as Fig.12As shown, since it has the same structure as the server 50, the description of each hardware structure is omitted. In addition to having the same structure as the server 50, the communication terminal 90 also has a short-distance communication circuit 917 and an antenna 917a of the short-distance communication circuit 917. The short-distance communication circuit 917 is a communication circuit such as NFC, Bluetooth, or Wi-Fi.

[0138] In addition, each of the above programs can also be recorded in a computer-readable recording medium in an installable or executable form and circulated. Examples of recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray Disc, SD card, USB memory, etc. In addition, the recording medium can be provided as a program product to domestic or foreign countries. For example, the server 50 implements the image processing method of the present invention by executing the program of the present invention.

[0139] ●Functional structure

[0140] Next, use Fig.13 The functional structure of the image processing system according to the embodiment will be described. Fig.13 is a diagram showing an example of a functional structure of an image processing system. Fig.13 express Figure 1 The portion of the device or terminal shown that is associated with the processing or action described later.

[0141] ○ Functional structure of the photographic device○

[0142] First, use Fig.13 The functional structure of the photographing device 10 is described. The photographing device 10 includes a transmitting and receiving unit 11, an operation receiving unit 12, a photographing control unit 13, a moving image photographing unit 14, a still image photographing unit 15, an image processing unit 16, a data management unit 17, and a storage / reading unit 19. These units are Fig.11 The functions or mechanisms implemented by any of the components shown in the figure are implemented by operating according to the command from the CPU 111 of the program for the photographic device which is developed from the SRAM 113 to the DRAM 114. Fig.11 The ROM 112, SRAM 113 and DRAM 114 shown in the figure constitute a storage unit 1000. The storage unit 1000 stores the GUID (Globally Unique Identifier) ​​of the device.

[0143] The transceiver 11 is an example of a transmitting mechanism and a receiving mechanism, and is mainly realized by processing of the CPU 111, and communicates various data or information with other devices or terminals. Moreover, the transceiver 11 uses the network I / F 121 to perform data communication with other devices or terminals via the communication network 100.

[0144] The operation accepting unit 12 is mainly realized by the processing of the CPU 111 on the operation unit 115 , and accepts various selections and inputs from the user who is a photographer.

[0145] The photography control unit 13 is mainly implemented by the processing of the imaging unit 101, the image processing unit 104, and the imaging control unit 105 by the CPU 111, and captures a subject such as a landscape to obtain photographic image data. For example, the photography control unit 13 performs time-division switching between the moving image capture by the moving image capture unit 14 and the still image capture by the still image capture unit 15.

[0146] The dynamic image shooting unit 14 is mainly realized by the processing of the camera unit 101, the image processing unit 104 and the camera control unit 105 by the CPU 111, and performs dynamic image shooting of the photographic device 10. The dynamic image shooting unit 14 shoots dynamic images while, for example, moving in a structure such as real estate as a predetermined location or in a space where a structure is arranged. The dynamic image shooting unit 14 shoots dynamic images at a low resolution in continuous frames while the photographer who keeps the photographic device 10 moves, and stores the shot image data in the storage unit 1000. The dynamic image shooting unit 14 shoots dynamic images when, for example, the photographer who keeps the photographic device 10 moves from a first location in the real estate as a predetermined location to a second location.

[0147] The static image photographing section 15 is mainly realized by processing the camera unit 101, and photographs a subject such as a landscape, and photographs a static image of the photographing device 10. The static image photographing section 15, for example, photographs a plurality of static images formed at different photographing positions in a structure such as a real estate or a space in which the structure is arranged as a specified location. The static image photographing section 15, for example, photographs a static image (photo) with a higher resolution than the dynamic image photographed by the dynamic image photographing section 14, and stores the photographed image data in the storage section 1000. The static image photographed by the static image photographing section 15 may be an image of one frame, or an HDR (high dynamic range) image synthesized from a plurality of images. In addition, dynamic image photography is performed while moving in a specified location, and static image photography is performed while stationary in a specified location.

[0148] Here, the photography specifications required for dynamic image photography and static image photography are different according to their respective photography purposes. In the case of dynamic image photography, continuous images (continuous frames) are obtained at a high frame rate for position estimation. On the other hand, high resolution and color layers of the photographic image are not necessary, and low resolution and grayscale are sufficient. In addition, in the case of static image photography, it is not necessary to obtain continuous frames. On the other hand, high resolution, color information (RGB), and high dynamic range shooting are required. The static image taken by the static image photography unit 15 is preferably a high resolution of 4K or above for the purpose of obtaining a reading image. On the other hand, the dynamic image taken by the dynamic image photography unit 14 is an image used for position estimation. It is sufficient as long as the subject captured in the dynamic image can be identified, so it is an image with a lower resolution than the static image. The dynamic image can be, for example, a resolution of about 480p or less. The photographic device 10 can reduce the overall data volume in the tour photography by shooting low-resolution dynamic images.

[0149] The image processing unit 16 is mainly realized by processing by the CPU 111 , and performs various image processing on the still image captured by the still image capturing unit 15 and the moving image captured by the moving image capturing unit 14 .

[0150] The data management unit 17 is an example of a determination unit, is mainly realized by the processing of the CPU 111, and performs processing for managing various types of device information.

[0151] The storage / reading unit 19 is an example of a setting mechanism, which is mainly realized by the processing of the CPU 111, and stores various data (or information) in the storage unit 1000 or reads various data (or information) from the storage unit 1000. In addition, the image data captured by the dynamic image capturing unit 14 and the still image capturing unit 15 are stored in the storage unit 1000. The shooting time of the captured image is associated with the image data stored in the storage unit 1000 as metadata.

[0152] The storage unit 1000 includes a setting information management DB 1001, an image management DB 1002, and a device information management DB 1003. The setting information management DB 1001 includes a setting information management table described below, and stores and manages setting information of image processing performed by the image processing unit 16 on still images or moving images.

[0153] The image management DB 1002 stores and manages the photographic images taken by the photographing device 10 and the processed images obtained by performing image processing on the photographic images by the image processing unit 16 in association with the image IDs. The photographic images and the processed images may be stored in a storage medium such as an SD memory card as the external medium.

[0154] The device information management DB 1003 stores and manages various device information.

[0155] ○Server function configuration○

[0156] First, use Fig.13 The functional structure of the server 50 is described. The server 50 includes a transmission and reception unit 51, a reception unit 52, an authentication unit 53, a data processing unit 54, a determination unit 55, an image processing unit 57, and a storage / reading unit 59. Fig.12 The functions or mechanisms implemented by any of the components shown in the figure are implemented by operating according to the instructions from the CPU 501 of the server program developed from the HD 504 to the RAM 503. Fig.12 The storage unit 5000 is composed of the ROM 502 , RAM 503 , and HD 504 shown.

[0157] The transmission and reception unit 51 is an example of a transmission mechanism and a reception mechanism, and is mainly realized by the processing of the network I / F 509 by the CPU 501, and transmits and receives various data or information with other devices or terminals via the communication network 100. The transmission and reception unit 51 receives (acquires) a moving image captured by the imaging device 10 from, for example, the imaging device 10 or the communication terminal 90. In addition, the transmission and reception unit 51 receives (acquires) a still image captured by the imaging device 10 from, for example, the imaging device 10 or the communication terminal 90.

[0158] The receiving unit 52 is mainly implemented by the CPU 501 processing the keyboard 511 and the pointing device 512 , and receives various selections and inputs from the user.

[0159] The authentication unit 53 is mainly realized by processing of the CPU 501 and performs various authentications.

[0160] The data processing unit 54 is mainly realized by processing of the CPU 501 and performs data processing of various device information.

[0161] The determination unit 55 is realized by the processing of the CPU 501 and performs various determination processes.

[0162] The image processing unit 57 is realized by the processing of the CPU 501 and performs various image processing on a still image or a moving image.

[0163] The storage / reading unit 59 is an example of a setting means, is mainly realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000 .

[0164] The storage unit 5000 includes a setting information management DB 5001, an image management DB 5002, a device information management DB 5003, and an account management DB 5004. The setting information management DB 5001 includes a setting information management table described below, and stores and manages setting information of image processing performed by the image processing unit 57 on still images or moving images.

[0165] The image management DB 5002 stores and manages processed images processed by the photographing device 10 or the communication terminal 90 and received from the photographing device 10 or the communication terminal 90 and processed images obtained by performing image processing by the image processing unit 57 , respectively, in association with image IDs.

[0166] The processed image processed by the image processing unit 57 may be sent to the photographing device 10 or the communication terminal 90 and stored in a storage medium such as the image management DB 1002 or 9002 or an SD memory card on the photographing device 10 or the communication terminal 90 side.

[0167] The device information management DB 5003 stores and manages various device information. The device information is an example of information other than images. The account management DB 5004 stores and manages account information such as user IDs, management information for managing the imaging device 10, and the like.

[0168] ○Functional structure of communication terminal○

[0169] Next, use Fig.13 The functional structure of the communication terminal 90 is described. The communication terminal 90 includes a transmission and reception unit 91, a reception unit 92, a display control unit 93, an image processing unit 94, a data control unit 95, an application function unit 96, an authentication unit 97, and a storage / reading unit 99. These units are Fig.13 The functions or mechanisms implemented by any of the components shown in the figure are implemented by operating in accordance with the instructions from the CPU 901 of the communication terminal program developed from the HD 904 to the RAM 903. Fig.12 The ROM 902, RAM 903, and HD 904 shown in the figure constitute a storage unit 9000. The storage unit 9000 stores various application programs.

[0170] The transmission and reception unit 91 is an example of a transmission unit and a reception unit, is mainly realized by the processing of the network I / F 909 by the CPU 901 , and transmits and receives various data or information with other devices or terminals via the communication network 100 .

[0171] The receiving unit 92 is an example of a setting means, is mainly realized by the processing of the keyboard 911 and the pointing device 912 by the CPU 901, and receives various selections and inputs from the user.

[0172] The display control unit 93 is mainly realized by the processing of the CPU 901, and displays various images and characters on the display 906. The display control unit 93 accesses the server 50 using, for example, a web browser or a dedicated application, and displays an image corresponding to data sent from the server 50 on the display 906.

[0173] The image processing unit 94 is mainly realized by processing of the CPU 901 , and performs various image processing on still images or moving images.

[0174] The data control unit 95 is mainly realized by processing of the CPU 901 and controls various data.

[0175] The application function unit 96 is mainly realized by processing by the CPU 901 , and executes functions of various applications stored in the storage unit 9000 .

[0176] Specifically, the application function unit 96 performs the function of the sending and receiving-related application that mediates the sending and receiving of information between the photographic device 10 and the server 50, and cooperates with the sending and receiving unit 91, the data control unit 95, the sending and receiving unit 11 of the photographic device 10, the data management unit 17, the sending and receiving unit 51 of the server 50, and the data processing unit 54 to control the data and information that can be communicated between the photographic device 10, the communication terminal 90, and the server 50.

[0177] The authentication unit 97 is mainly implemented by the processing of the CPU 901, and performs various authentications.

[0178] The storage / reading unit 99 is mainly realized by processing by the CPU 901 , and stores various data (or information) in the storage unit 9000 , and reads various data (or information) from the storage unit 9000 .

[0179] The storage unit 9000 has a setting information management DB 9001 and an image management DB 9002. The setting information management DB 9001 is composed of a setting information management table described later, and stores and manages setting information and the like of image processing performed by the image processing unit 94 on still images or moving images.

[0180] The image management DB 9002 stores and manages photographic images captured by and received from the photographic device 10 and processed images obtained by performing image processing on the photographic images by the image processing unit 94, in association with image IDs. The photographic images and processed images may also be stored in a storage medium such as an SD memory card as an external medium.

[0181] Fig.14 This is a diagram showing another example of the functional structure of the communication terminal.

[0182] Fig.14 The communication terminal 90A shown in (A) and Fig.13 The communication terminal 90 shown is the same as that shown in FIG. 9 , and includes a transmission and reception unit 91, a reception unit 92, a display control unit 93, a data control unit 95, an application function unit 96, and a storage / reading unit 99, but does not include an image processing unit 94 and an authentication unit 97. In addition, an image management DB 9002 is constructed in the storage unit 9000, but a setting information management DB 9001 is not constructed.

[0183] Fig.14 The communication terminal 90B shown in (B) and Fig.13 The communication terminal 90 shown is the same as that shown in FIG. 9 , and includes a transmission and reception unit 91, a reception unit 92, a data control unit 95, an application function unit 96, and a storage / reading unit 99, but does not include a display control unit 93, an image processing unit 94, and an authentication unit 97. In addition, an image management DB 9002 is constructed in the storage unit 9000, but a setting information management DB 9001 is not constructed.

[0184] Communication terminals 90A and 90B are examples of communication terminals on which transmission and reception-related applications developed by other companies are installed.

[0185] Fig.15 This is a conceptual diagram showing an example of the account management table involved in this embodiment.

[0186] Fig.15 Indicates that the Fig.13 The account management table of the account management DB 5004 constructed in the storage unit 5000 of the server 50 is shown.

[0187] In the composition Fig.15 In the account management table of the account management DB 5004 shown in (A), a team ID for identifying a team using an application, a user ID for identifying a user belonging to the team, and an email address and a password of the user are managed in association with an application ID.

[0188] The application ID is information for identifying the type of application related to transmission and reception stored in storage unit 9000 of communication terminal 90 .

[0189] Here, the application function unit 96 of the communication terminal 90 performs the function of the sending and receiving related application that mediates the sending and receiving of information between the photographic device 10 and the server 50, and controls the data and information that can be communicated between the photographic device 10, the communication terminal 90 and the server 90. Therefore, it can be said that the data and information that can be communicated are determined according to the type of the sending and receiving related application.

[0190] The application ID "RICOH-APP" is an example of an application developed by our company. As described later, many types of information can be sent and received between the imaging device 10 and the server 90 of our system.

[0191] In the composition Fig.15 In the account management table of the account management DB5004 shown in (B), the application ID, team ID and user ID are associated to manage the virtual device ID for identifying the photographic device 10, the serial number of the photographic device 10, the image data (the path to the image management DB5002) and the device information (the path to the device information management DB5003).

[0192] In the composition Fig.15 In the account management table of the account management DB 5004 shown in (C), the virtual device ID, the serial number of the imaging device 10, the image data, and the device information are managed in association with the application ID and the group ID identifying the group using the application.

[0193] The application ID "OTHER-APP" is an example of an application developed by another company and does not completely correspond to the company's system. As described later, the types of information that can be sent and received between the photographing device 10 and the server 90 of the company's system are relatively small.

[0194] In the account management table constituting the account management DB5004, except Fig.15 In addition to the information shown, communication terminals 90, 90A, and 90B manage management information and image identification information for managing imaging device 10. Image identification information is information other than images or an example of second other information different from other information, and management information is an example of second other information.

[0195] Fig.16 This is a conceptual diagram showing an example of a setting information management table according to the present embodiment.

[0196] Fig.16 (A) indicates that Fig.13 The settings information management table of the settings information management DB 1001 constructed in the storage unit 1000 of the imaging device 10 and the settings information management DB 5001 constructed in the storage unit 5000 of the server 50 is shown.

[0197] In the setting information management table constituting the setting information management DB 1001 and the setting information management DB 5001, image processing setting information including image processing information on the imaging device side and image processing information on the server side is managed in association with the team ID and the virtual device ID.

[0198] The image processing information on the photographing device side is an example of first processing information indicating a first image processing performed on the photographing device 10 side as the transmission source device, and the image processing information on the server side is an example of second processing information indicating a second image processing performed on the server 50 side as the transmission destination device. The first image processing performed by the photographing device 10 may be image processing with a relatively small load, and the second image processing performed by the server 50 may be image processing with a relatively large load.

[0199] In addition, in addition to the connection processing of connecting the images respectively obtained by the camera elements 103a and 103b described later to generate a full celestial spherical image, the first image processing and the second image processing can also be applied to the present embodiment as long as it is processing of the image, for example, zenith correction for adjusting the top and bottom of the image to a vertical upward direction or a downward direction, hand shake correction for correcting image shake caused by hand shake, etc., super-resolution processing for improving the resolution of the image, spot correction for correcting the light spot in which strong light is reflected by the lens, etc. and makes part or the whole of the image white, object detection processing for detecting specific objects such as faces or people, marks for object detection, denoising processing for removing or reducing noise generated in the image, etc.

[0200] In addition, there are various image processings other than these, but as this embodiment, there may be image processing executed only in the photographing device 10 and the above-mentioned image processing as an example. As an example of image processing executed only in the photographing device 10, there is shading correction or color correction processed in the ISP. The image captured by the photographing device 10 is compressed by the photographing device 10 before being sent to the server 50. Therefore, the image processing that must be processed before the image compression is equivalent to the image processing executed only in the photographing device 10. In other words, the image processing can be processed as the first image processing or the second image processing even after the image compression.

[0201] That is, the image processing described in this embodiment is roughly divided into two types: the image processing inherent to the imaging device side executed only in the imaging device 10, and the first image processing and the second image processing executed on the imaging device side or the server side. In addition, the image processing shared between the imaging device side and the server side as described later will be described with the first image processing and the second image processing as the object.

[0202] Fig.16 (B) indicates that the Fig.13 The configuration information management DB 9001 constructed in the storage unit 9000 of the communication terminal 90 and the configuration information management table of the configuration information management DB 5001 constructed in the storage unit 5000 of the server 50 are shown.

[0203] In the setting information management table constituting the setting information management DB 9001 and the setting information management DB 5001, image processing setting information including communication terminal side processing information and server side image processing information is managed in association with the team ID and the virtual device ID.

[0204] The communication terminal side image processing information is an example of first processing information indicating a first image processing performed on the communication terminal 90 side as the transmission source device, and the server side image processing information is an example of second processing information indicating a second image processing performed on the server 50 side as the transmission destination device. The first image processing performed by the communication terminal 90 may be image processing with a relatively small load, and the second image processing performed by the server 50 may be image processing with a relatively large load.

[0205] Fig.16 (C) indicates that the Fig.13 The setting information management table of the setting information management DB 1001 constructed in the storage unit 1000 of the imaging device 10 and the setting information management DB 9001 constructed in the storage unit 9000 of the communication terminal 90 is shown.

[0206] In the setting information management table constituting the setting information management DB 1001 and the setting information management DB 9001, image processing setting information including image processing information on the imaging device side and image processing information on the communication terminal side is managed in association with the team ID and the virtual device ID.

[0207] The image processing information on the photographing device side is an example of first processing information indicating a first image processing performed on the photographing device 10 side, and the image processing information on the communication terminal side is an example of second processing information indicating a second image processing performed on the communication terminal 90 side. The first image processing performed by the photographing device 10 may be image processing with a relatively small load, and the second image processing performed by the communication terminal 90 may be image processing with a relatively large load.

[0208] Fig.16 The image processing setting information in the setting information management table shown in (A) is set corresponding to the transmission path for directly transmitting the image from the imaging device 10 to the server 50.

[0209] Fig.16 (B) and Fig.16 The image processing setting information in the setting information management table shown in (C) is set corresponding to the transmission path of the image from the photographing device 10 to the server 50 via the communication terminal 90. In this case, the image processing setting information may also include all of the processing information on the photographing device side, the image processing information on the communication terminal side, and the image processing information on the server side.

[0210] Fig.17It is a conceptual diagram showing an example of a device information management table involved in this embodiment. Fig.17 Indicates that the Fig.13 The device information management table of the device information management DB 1003 constructed in the storage unit 1000 of the imaging device 10 is shown.

[0211] In the composition Fig.17 In the device information management table of the device information management DB 1003 shown in FIG. 1004 , the types of device information that can be sent are managed according to whether the sending destination is the server 50 or the sending destination is the communication terminal 90. In addition, when the sending destination is the communication terminal 90, Fig.15 The types of device information that can be sent are managed by associating it with the application ID described in .

[0212] Fig.18 This is a sequence diagram showing an example of registration processing by direct transmission by the server according to the present embodiment.

[0213] The receiving unit 92 of the communication terminal 90 receives the input information by the user inputting the user ID and password (step S1), and the authentication unit 97 authenticates the information received by the receiving unit 92 (step S2), for example, by requesting authentication from the cloud service through a smartphone application.

[0214] The transmission and reception unit 91 transmits the user ID authenticated in step S2 to the server 50 , and the transmission and reception unit 51 of the server 50 receives the user ID transmitted from the communication terminal 90 (step S3 ).

[0215] The authentication unit 53 refers to the account management DB 5004 via the storage / reading unit 59 , and authenticates the user ID received in step S3 (step S4 ).

[0216] The receiving unit 92 of the communication terminal 90 receives information related to the registration request by inputting information of the user requesting to register the sending path directly sent from the photographic device 10 to the server 50, and the storage / reading unit 99 reads the team ID stored in the storage unit 9000 corresponding to the user ID (step S5).

[0217] The sending and receiving unit 91 sends the information indicating the sending path received in step S5 and the team ID read in step S5 to the photographic device 10. The sending and receiving unit 11 of the photographic device 10 receives the information indicating the sending path and the team ID sent from the communication terminal 90. The data management unit 17 stores the information indicating the received sending path in the storage unit 1000 through the storage / reading unit 19 (step S6).

[0218] The data management unit 17 reads the application ID and the sequence number of the photographic device 10 stored corresponding to the communication terminal 90 from the storage unit 1000 through the storage / reading unit 19, and the sending and receiving unit 11 sends the application ID, sequence number and team ID to the server 50. The sending and receiving unit 51 of the server 50 receives the application ID, sequence number and team ID sent from the photographic device 10 (step S7).

[0219] The authentication unit 53 refers to the account management DB5004 to confirm whether the various information received in step S7 has been registered. If not registered, a virtual device ID is assigned corresponding to the sequence No. (step S8). The data processing unit 54 stores the various information received in step S7 and the virtual device ID in the account management DB5004 correspondingly through the storage / reading unit 59 (step S9).

[0220] The transceiver 51 transmits the virtual device ID assigned in step S8 to the photographing device 10, and the transceiver 11 of the photographing device 10 receives the virtual device ID transmitted from the server 50 (step S10). Thus, a transmission path directly transmitted from the photographing device 10 to the server 50, that is, a second transmission path not passing through the communication terminal 90, is registered.

[0221] In step S8 , when it is confirmed that the various information received in step S7 has been registered, in step S10 , the transceiver 51 transmits an error message indicating registration to the photographing device 10 , and the transceiver 11 of the photographing device 10 receives the error message transmitted from the server 50 .

[0222] Fig.19 (A) is a sequence diagram showing an example of direct transmission of an image by a server according to the present embodiment.

[0223] The data management unit 17 of the photographic device 10 reads the image from the image management DB 1002 through the storage / reading unit 19, and the sending and receiving unit 11 sends the application ID, the virtual device ID, the image data representing the image, and the image identification information to the server 50. The sending and receiving unit 51 of the server 50 receives the application ID, the virtual device ID, the image data, and the image identification information sent from the photographic device 10 (step S21).

[0224] The authentication unit 53 refers to the account management DB5004 to authenticate the various information received in step S21 (step S22), and the data processing unit 54 stores the image related to the image data received in step S21 in the image management DB5002 through the storage / reading unit 59, and stores the path pointing to the image management DB5002 in the account management DB5004 corresponding to the image identification information, the application ID received in step S21, and the virtual device ID (step S23).

[0225] When image processing is performed on the image related to the image data received in step S21 , in step S23 , the data processing unit 54 stores the processed image on which the image processing is performed in the image management DB 5002 via the storage / reading unit 59 .

[0226] Fig.19 (B) is a sequence diagram showing an example of direct transmission of device information by the server according to the present embodiment.

[0227] The data management unit 17 of the photographing device 10 refers to the Fig.17 The device information management table shown reads the device information that the server 50 can receive from the device information management DB1003, and the sending and receiving unit 11 sends the device information read from the application ID, virtual device ID and storage unit 1000 to the server 50. The sending and receiving unit 51 of the server 50 receives the application ID, virtual device ID and device information sent from the photographic device 10 (step S31).

[0228] The authentication unit 53 refers to the account management DB5004 to authenticate the various information received in step S31 (step S32), and the data processing unit 54 stores the device information received in step S31 in the device information management DB5003 through the storage / reading unit 59, and stores the path pointing to the device information management DB5003 in the account management DB5004 corresponding to the application ID and virtual device ID received in step S31 (step S33).

[0229] Fig. 20 This is a sequence diagram showing an example of the registration process of the sending and receiving agent involved in this embodiment.

[0230] The receiving unit 92 of the communication terminal 90 receives input information by the user inputting the user ID and password (step S41 ), and the authentication unit 97 authenticates the information received by the receiving unit 92 (step S42 ).

[0231] The transmission and reception unit 91 transmits the user ID authenticated in step S42 to the server 50 , and the transmission and reception unit 51 of the server 50 receives the user ID transmitted from the communication terminal 90 (step S43 ).

[0232] The authentication unit 53 refers to the account management DB 5004 via the storage / reading unit 59 , and authenticates the user ID received in step S43 (step S44 ).

[0233] The receiving unit 92 of the communication terminal 90 receives information related to the registration request by inputting the registration information of the sending path sent from the photographic device 10 to the server 50 via the communication terminal 90, and the storage / reading unit 99 reads the team ID stored in the storage unit 9000 corresponding to the user ID (step S45).

[0234] The sending and receiving unit 91 sends the information indicating the sending path and the acquisition request indicating the sequence No. of the photographic device 10 received in step S45 to the photographic device 10. The sending and receiving unit 11 of the photographic device 10 receives the information indicating the sending path and the acquisition request sent from the communication terminal 90. The data management unit 17 stores the received information indicating the sending path in the storage unit 1000 through the storage / reading unit 19 (step S46).

[0235] The data management unit 17 reads the sequence number of the photographic device 10 from the storage unit 1000 via the storage / reading unit 19, the transceiver unit 11 sends the sequence number to the communication terminal 90, and the transceiver unit 91 of the communication terminal 90 receives the sequence number sent from the photographic device 10 (step S47).

[0236] The sending and receiving unit 91 sends the application ID, the sequence number received in step S47, the team ID and the user ID read in step S45 to the server 50, and the sending and receiving unit 51 of the server 50 receives the application ID, sequence number, team ID and user ID sent from the communication terminal 90 (step S48).

[0237] The authentication unit 53 refers to the account management DB5004 to confirm whether the various information received in step S48 has been registered. If not registered, a virtual device ID is assigned corresponding to the sequence No. (step S49). The data processing unit 54 stores the various information received in step S48 and the virtual device ID in the account management DB5004 correspondingly through the storage / reading unit 59 (step S50).

[0238] The transceiver 51 transmits the virtual device ID assigned in step S49 to the photographing apparatus 10, and the transceiver 11 of the photographing apparatus 10 receives the virtual device ID transmitted from the server 50 (step S51). Thus, the first transmission path transmitted from the photographing apparatus 10 to the server 50 via the communication terminal 90 is registered.

[0239] In step S49 , when it is confirmed that the various information received in step S48 has been registered, in step S51 , the transceiver 51 sends an error message indicating that the information has been registered to the photographic device 10 , and the transceiver 11 of the photographic device 10 receives the error message sent from the server 50 .

[0240] Fig.21 This is a sequence diagram showing an example of the image sending and receiving intermediary processing involved in this embodiment.

[0241] The receiving unit 92 of the communication terminal 90 receives input information by the user inputting the user ID and password (step S61 ), and the authentication unit 97 authenticates the information received by the receiving unit 92 (step S62 ).

[0242] The transmission and reception unit 91 transmits the user ID authenticated in step S62 to the server 50 , and the transmission and reception unit 51 of the server 50 receives the user ID transmitted from the communication terminal 90 (step S63 ).

[0243] The authentication unit 53 refers to the account management DB 5004 via the storage / reading unit 59 , and authenticates the user ID received in step S63 (step S64 ).

[0244] The receiving unit 92 of the communication terminal 90 receives information on the upload request in response to the user inputting information requesting the upload of an image (step S65 ).

[0245] The transceiver 91 transmits an image acquisition request to the photographing device 10, and the transceiver 11 of the photographing device 10 receives the acquisition request transmitted from the communication terminal 90 (step S66).

[0246] The data management unit 17 reads the image from the image management DB 1002 through the storage / reading unit 19, and the sending and receiving unit 11 sends the virtual device ID, image data representing the read image, and image identification information to the communication terminal 90, and the sending and receiving unit 91 of the communication terminal 90 receives the virtual device ID, image data, and image identification information (step S67).

[0247] The data control unit 95 reads the management information of the photographic device 10 from the storage unit 9000 through the storage / reading unit 99, and the sending and receiving unit 91 sends the application ID, virtual device ID, user ID, image data, image identification information and management information to the server 50. The sending and receiving unit 51 of the server 50 receives the application ID, virtual device ID, user ID, image data, image identification information and management information sent from the communication terminal 90 (step S68).

[0248] The authentication unit 53 refers to the account management DB5004 to authenticate the various information received in step S68 (step S69), and the data processing unit 54 stores the image related to the image data received in step S68 in the image management DB5002 through the storage / reading unit 59, and stores the path pointing to the image management DB5002 in the account management DB5004 corresponding to the management information and image identification information, the application ID received in step S68, and the virtual device ID (step S70).

[0249] When image processing is performed on the image related to the image data received in step S68 , in step S70 , the data processing unit 54 stores the processed image on which the image processing is performed in the image management DB 5002 via the storage / reading unit 59 .

[0250] Fig. 22 This is a sequence diagram showing an example of the intermediary processing of sending and receiving device information involved in this embodiment.

[0251] The receiving unit 92 of the communication terminal 90 receives input information by the user inputting the user ID and password (step S81 ), and the authentication unit 97 authenticates the information received by the receiving unit 92 (step S82 ).

[0252] The transmission and reception unit 91 transmits the user ID authenticated in step S82 to the server 50 , and the transmission and reception unit 51 of the server 50 receives the user ID transmitted from the communication terminal 90 (step S83 ).

[0253] The authentication unit 53 refers to the account management DB 5004 via the storage / reading unit 59 , and authenticates the user ID received in step S83 (step S84 ).

[0254] The receiving unit 92 of the communication terminal 90 receives information related to the upload request when the user inputs information requesting the upload of the device information (step S85).

[0255] The transceiver 91 transmits a request for obtaining device information to the photographing apparatus 10 , and the transceiver 11 of the photographing apparatus 10 receives the request for obtaining device information transmitted from the communication terminal 90 (step S86 ).

[0256] The data management unit 17 refers to the storage / reading unit 19 Fig.17 The device information management table shown reads the device information that the communication terminal 90 can receive from the storage unit 1000, the sending and receiving unit 11 sends the virtual device ID and the device information read from the storage unit 1000 to the communication terminal 90, and the sending and receiving unit 91 of the communication terminal 90 receives the virtual device ID and device information sent from the photographic device 10 (step S87).

[0257] The data control unit 95 reads the management information of the photographic device 10 from the storage unit 9000 through the storage / reading unit 99, and the sending and receiving unit 91 sends the application ID, virtual device ID, user ID, device information and management information to the server 50. The sending and receiving unit 51 of the server 50 receives the application ID, virtual device ID, device information and management information sent from the communication terminal 90 (step S88).

[0258] The authentication unit 53 refers to the account management DB5004 to authenticate the various information received in step S88 (step S89), and the data processing unit 54 stores the device information received in step S88 in the device information management DB5003 through the storage / reading unit 59, and stores the path pointing to the device information management DB5003 in the account management DB5004 corresponding to the management information, the application ID received in step S88, and the virtual device ID (step S90).

[0259] Fig.23 This is a sequence diagram showing an example of the registration process of another sending and receiving agent involved in this embodiment.

[0260] The receiving unit 92 of the communication terminal 90 receives information related to the registration request by inputting the information of the sending path sent from the photographic device 10 to the server 50 via the communication terminals 90A and 90B in which the user requests to register, and the storage / reading unit 99 reads the group ID stored in the storage unit 9000 corresponding to the user ID (step S101).

[0261] The sending and receiving unit 91 sends the information indicating the sending path received in step S101 and the acquisition request for the sequence No. of the photographic device 10 to the photographic device 10, the sending and receiving unit 11 of the photographic device 10 receives the information indicating the sending path and the acquisition request sent from the communication terminal 90, and the data management unit 17 stores the received information indicating the sending path in the storage unit 1000 through the storage / reading unit 19 (step S102).

[0262] The data management unit 17 reads the sequence number of the photographic device 10 from the storage unit 1000 via the storage / reading unit 19, the sending and receiving unit 11 sends the sequence number to the communication terminal 90, and the sending and receiving unit 91 of the communication terminals 90A and 90B receives the sequence number sent from the photographic device 10 (step S103).

[0263] The transmission and reception unit 91 transmits the application ID, the sequence number, and the group ID received in step S103 to the server 50 , and the transmission and reception unit 51 of the server 50 receives the application ID, sequence number, and group ID transmitted from the communication terminals 90A and 90B (step S104 ).

[0264] The authentication unit 53 refers to the account management DB5004 to confirm whether the various information received in step S104 has been registered. If not registered, a virtual device ID is assigned corresponding to the sequence No. (step S105). The data processing unit 54 stores the various information received in step S104 and the virtual device ID in the account management DB5004 correspondingly through the storage / reading unit 59 (step S106).

[0265] The transmission and reception unit 51 transmits the virtual device ID assigned in step S105 to the photographing apparatus 10 , and the transmission and reception unit 11 of the photographing apparatus 10 receives the virtual device ID transmitted from the server 50 (step S107 ).

[0266] In step S105 , when it is confirmed that the various information received in step S104 has been registered, in step S107 , the transceiver 51 transmits an error message indicating registration to the photographic device 10 , and the transceiver 11 of the photographic device 10 receives the error message transmitted from the server 50 .

[0267] Fig.24 This is a sequence diagram showing an example of another image sending and receiving intermediary processing involved in this embodiment.

[0268] The receiving unit 92 of the communication terminal 90A or 90B receives information on the upload request when the user inputs information requesting the upload of an image (step S111 ).

[0269] The transceiver 91 transmits an image acquisition request to the photographing device 10 , and the transceiver 11 of the photographing device 10 receives the acquisition request transmitted from the communication terminal 90 (step S112 ).

[0270] The data management unit 17 reads the image from the image management DB 1002 via the storage / reading unit 19, and the sending / receiving unit 11 sends the virtual device ID and the image data representing the read image to the communication terminals 90A and 90B, and the sending / receiving unit 91 of the communication terminals 90A and 90B receives the virtual device ID and the image data (step S113). The data management unit 17 determines that the image recognition information cannot be received in the application related to sending / receiving possessed by the communication terminals 90A and 90B, and Fig.21 Unlike step 67, the transmission / reception unit 11 does not transmit the image recognition information to the communication terminals 90A and 90B.

[0271] The sending and receiving unit 91 sends the application ID, virtual device ID, group ID and image data to the server 50, and the sending and receiving unit 51 of the server 50 receives the application ID, virtual device ID, group ID and image data sent from the communication terminals 90A and 90B (step S114). The data control unit 95 controls the application ID, virtual device ID, group ID and image data according to the application related to the sending and receiving. Fig.21 Unlike step 68, the sending and receiving unit 91 does not send the management information to the server 50.

[0272] The authentication unit 53 refers to the account management DB5004 to authenticate the various information received in step S114 (step S115), and the data processing unit 54 stores the image related to the image data received in step S114 in the image management DB5002 through the storage / reading unit 59, and stores the path pointing to the image management DB5002 in the account management DB5004 corresponding to the application ID, group ID and virtual device ID received in step S114 (step S116).

[0273] When image processing is performed on the image related to the image data received in step S114 , in step S116 , the data processing unit 54 causes the storage / reading unit 59 to store the processed image on which the image processing is performed in the image management DB 5002 .

[0274] Fig.25 This is a sequence diagram showing an example of another sending and receiving intermediary processing of device information involved in this embodiment.

[0275] The reception unit 92 of the communication terminal 90A or 90B receives information related to the upload request when the user inputs information requesting the upload of the device information (step S121 ).

[0276] The transceiver 91 transmits a request for obtaining device information to the photographing apparatus 10 , and the transceiver 11 of the photographing apparatus 10 receives the request for obtaining device information transmitted from the communication terminals 90A and 90B (step S122 ).

[0277] The data management unit 17 refers to the storage / reading unit 19 Fig.17 The device information management table shown reads the device information that can be received by the communication terminals 90A and 90B from the storage unit 1000, the sending and receiving unit 11 sends the virtual device ID and the device information read from the storage unit 1000 to the communication terminals 90A and 90B, and the sending and receiving unit 91 of the communication terminals 90A and 90B receives the virtual device ID and device information sent from the photographic device 10 (step S123).

[0278] The sending and receiving unit 91 sends the application ID, virtual device ID, user ID and device information to the server 50, and the sending and receiving unit 51 of the server 50 receives the application ID, virtual device ID, user ID and device information sent from the communication terminals 90A and 90B (step S124). The data control unit 95 communicates with the application related to the sending and receiving according to the application related to the sending and receiving. Fig. 22 Unlike step 88, the sending and receiving unit 91 does not send the management information to the server 50.

[0279] The authentication unit 53 refers to the account management DB5004 to authenticate the various information received in step S124 (step S125). The data processing unit 54 stores the device information received in step S124 in the device information management DB5003 through the storage / reading unit 59, and stores the path pointing to the device information management DB5003 in the account management DB5004 in correspondence with the application ID and virtual device ID received in step S124 (step S126).

[0280] Fig.26 This is a sequence diagram showing an example of the display process of the device information involved in this embodiment.

[0281] The receiving unit 92 of the communication terminal 90 receives information input by the user inputting the user ID and password (step S131 ), and the authentication unit 97 authenticates the information received by the receiving unit 92 (step S132 ).

[0282] The transmission and reception unit 91 transmits the user ID authenticated in step S132 to the server 50 , and the transmission and reception unit 51 of the server 50 receives the user ID transmitted from the communication terminal 90 (step S133 ).

[0283] The authentication unit 53 refers to the account management DB 5004 via the storage / reading unit 59 , and authenticates the user ID received in step S133 (step S134 ).

[0284] The receiving unit 92 of the communication terminal 90 receives information related to the display request when the user inputs information requesting display of the device information of the target imaging device 10 (step S135 ).

[0285] The transceiver 91 transmits the serial number of the photographing device 10 showing the application ID, user ID and device information to the server 50, and the transceiver 51 of the server 50 receives the application ID, user ID and serial number transmitted from the communication terminal 90 (step S136).

[0286] The authentication unit 53 refers to the account management DB5004 to authenticate the various information received in step S136 (step S137), and the data processing unit 54 reads the device information and management information of the photographic device 10 related to the sequence No. received in step S136 from the device information management DB5003 and the account management DB5004 through the storage / reading unit 59 (step S138).

[0287] The sending and receiving unit 51 sends the device information and management information read in step S138 to the communication terminal 90, the sending and receiving unit 91 of the communication terminal 90 receives the device information and management information sent from the server 50 (step S139), and the display control unit 93 displays the device information and management information received in step S139 on the display 906 (step S140).

[0288] Fig. 27 This is a sequence diagram showing another example of the display process of the device information involved in this embodiment.

[0289] The reception unit 92 of the communication terminal 90A receives information related to the display request when the user inputs information requesting display of the device information of the target imaging device 10 (step S151 ).

[0290] The transceiver 91 transmits the application ID, group ID, and sequence number of the photographing device 10 showing the device information to the server 50, and the transceiver 51 of the server 50 receives the application ID, group ID, and sequence number transmitted from the communication terminal 90 (step S152).

[0291] The authentication unit 53 refers to the account management DB5004 to authenticate the various information received in step S152 (step S153), and the data processing unit 54 reads the device information and management information of the photographic device 10 related to the sequence No. received in step S152 from the device information management DB5003 and the account management DB5004 through the storage / reading unit 59 (step S154).

[0292] The sending and receiving unit 51 sends the device information and management information read in step S154 to the communication terminal 90, the sending and receiving unit 91 of the communication terminal 90 receives the device information and management information sent from the server 50 (step S155), and the display control unit 93 displays the device information and management information received in step S155 on the display 906 (step S156).

[0293] Fig.28 This is a flowchart showing an example of image transmission processing in the imaging device involved in this embodiment.

[0294] The photography control unit 13 uses a predetermined condition as a trigger to capture a still image or a moving image through the still image photography unit 15 or the moving image photography unit 14, and the storage / reading unit 19 stores the photographic image representing the captured still image or moving image in the image management DB 1002 (step S161).

[0295] The data management unit 17 determines whether the sending path directly to the server 50 is registered by confirming the information representing the sending path stored in the storage unit 1000 by the storage / reading unit 19 (step S162), determines whether image processing is required based on the image processing setting information stored in the setting information management DB 1001, and determines whether image processing is possible based on the load information of the CPU 111 (step S163).

[0296] When image processing is necessary but impossible, the transmission and reception unit 11 transmits the photographic image captured in step S161 and image identification information indicating that the transmitted image is a photographic image to the server 50 (step S164).

[0297] When image processing is required and possible, the image processing unit 16 obtains the image from the storage / reading unit 19. Fig.16 The setting information management DB1001 shown in (A) reads the image processing information on the photographic device side, performs image processing represented by the image processing information on the photographic device side on the still image or moving image captured in step S161, and the storage / reading unit 19 stores the processed image representing the still image or moving image on which the image processing has been performed by the image processing unit 16 in the image management DB1002 in correspondence with the photographic image (step S165).

[0298] Here, Fig.16 The image processing information on the imaging device side in the setting information management DB 1001 shown in (A) is set in correspondence with the transmission path directly transmitted to the server 50 .

[0299] Next, the transmission and reception unit 11 transmits the processed image on which the image processing has been performed in step S165 and the image identification information indicating that the processed image and the image processing performed in step S165 are transmitted to the server 50 (step S166 ).

[0300] When it is determined in step S162 that the transmission route for direct transmission to the server 50 is not registered, the data management unit 17 determines whether the transmission and reception unit 11 has received an image acquisition request from the communication terminal 90 (step S167 ).

[0301] When receiving the image acquisition request, the data management unit 17 determines whether image processing is necessary based on the image processing setting information stored in the setting information management DB 1001 and determines whether image processing is possible based on the load information of the CPU 111 (step S168).

[0302] When image processing is unnecessary and impossible, the transmission and reception unit 11 transmits the photographic image captured in step S161 and the image identification information indicating that the transmitted image is the photographic image to the communication terminal 90 (step S169).

[0303] When image processing is necessary and possible, the image processing unit 16 obtains the image from the storage / reading unit 19. Fig.16 The setting information management DB1001 shown in (C) reads the image processing information on the photographic device side, performs image processing represented by the image processing information on the photographic device side on the still image or moving image captured in step S161, and the storage / reading unit 19 stores the processed image representing the still image or moving image on which the image processing has been performed by the image processing unit 16 in the image management DB1002 in correspondence with the photographic image (step S170).

[0304] Here, Fig.16 The imaging device side image processing information in the setting information management DB 1001 shown in (C) is set corresponding to the transmission path to the server 50 via the communication terminal 90 .

[0305] Next, the transmission / reception unit 11 transmits the processed image on which the image processing has been performed in step S170 and the image identification information indicating that the processed image and the image processing performed in step S170 are transmitted to the communication terminal 90 (step S171 ).

[0306] Fig.29 (A) is a flowchart showing an example of image transmission and reception processing in the communication terminal involved in this embodiment.

[0307] The data control unit 95 of the communication terminal 90 determines whether the transmission / reception unit 91 has received an image from the imaging device 10 (step S181 ).

[0308] When an image is received, the data control unit 95 determines whether image processing is required based on the image processing setting information stored in the setting information management DB 1001 and the image recognition information received together with the image in step S181, and determines whether image processing is possible based on the load information of the CPU 111 (step S182).

[0309] When image processing is not necessary and impossible, the sending and receiving unit 91 adds the management information of the photographic device 10 read from the storage unit 9000 by the storage / reading unit 99 to the image received in step S181 and the image identification information received together with the image, and sends it to the server 50 (step S183).

[0310] When image processing is necessary and possible, the image processing unit 94 obtains the image from the storage / reading unit 99. Fig.16 The setting information management DB9001 shown in (B) or (C) reads the image processing information on the communication terminal side, and performs image processing on the image received in step S181 based on the image processing information on the communication terminal side and the image recognition information received in step S181. The data control unit 95 stores the processed image on which the image processing has been performed by the image processing unit 94 in the image management DB9002 through the storage / reading unit 99 (step S184).

[0311] Specifically, the image recognition information received in step S181 is Fig.16 If the image processing information on the photographing device side shown in (C) is consistent, the image processing unit 94 executes Fig.16 Image processing on the communication terminal side shown in (C).

[0312] On the other hand, the image recognition information received in step S181 is Fig.16 If the image processing information on the photographing device side shown in (C) is inconsistent, the image processing unit 94 determines that the photographing device 10 cannot perform all the image processing that should be performed, and performs the image processing that the photographing device 10 cannot perform and Fig.16 Image processing on the communication terminal side shown in (C).

[0313] Next, the sending and receiving unit 11 adds the management information of the photographic device 10 to the processed image on which image processing has been performed in step S184, and the image identification information indicating that what is sent is the processed image and the image processing performed in step S184, and sends it to the server 50 in the same manner as step S183 (step S185).

[0314] Fig.29(B) is a flowchart showing an example of image reception processing in the server involved in this embodiment.

[0315] The determination unit 55 of the server 50 determines whether the transmission / reception unit 51 has received an image from the imaging device 10 or the communication terminal 90 (intermediary device) (step S191 ).

[0316] When an image is received, the judgment unit 55 judges whether image processing is required based on the server-side image processing information stored in the setting information management DB 5001 and the image recognition information received together with the image in step S191, and judges whether image processing is required based on the CPU 501 (step S192).

[0317] If image processing is necessary, the image processing unit 57 executes image processing based on the image received in step S191 , the image identification information received together with the image, and the server-side image processing information (step S193 ).

[0318] When the image processing has been performed, the data processing unit 54 stores the processed image on which the image processing has been performed in step S193 in the image management DB 5002 via the storage / reading unit 59. Furthermore, the data processing unit 54 stores the image identification information and management information of the processed image on which the image processing has been performed in step S193 in the account management DB 5004 via the storage / reading unit 59 in correspondence with the device information of the imaging device 10 or the communication terminal 90 (intermediary device) as the transmission destination device.

[0319] Fig.30 This is a flowchart showing an example of a process of sending device information in the photographing apparatus according to this embodiment.

[0320] The data management unit 17 determines whether the sending path directly sent to the server 50 is registered (step S201) by confirming the information representing the sending path stored in the storage unit 1000 by the storage / reading unit 19, and determines whether the start-up time of the photographic device 10 or the regularly specified sending time has been reached (step S202).

[0321] When the sending time arrives, the data management unit 17 reads the device information that the server 50 can receive from the device information management DB 1003 through the storage / reading unit 19, and the sending and receiving unit 11 sends the device information read from the storage unit 1000 together with the application ID and the virtual device ID to the server 50 (step S203).

[0322] When it is determined in step S201 that the transmission route for direct transmission to the server 50 is not registered, the data management unit 17 determines whether the transmission and reception unit 11 has received a request for obtaining device information from the communication terminal 90 (step S204 ).

[0323] When receiving the acquisition request, the data management unit 17 reads the device information receivable by the communication terminal 90 from the storage unit 1000 via the storage / reading unit 19, and the transmission and reception unit 11 transmits the device information read from the storage unit 1000 together with the virtual device ID to the communication terminal 90 (step S205).

[0324] Fig.31 FIG. 1 is a diagram showing an example of a display screen of device information involved in this embodiment, and shows Fig.26 Step 140 and Fig. 27 Display screen 2000 displayed on display 906 by display control unit 93 of communication terminal 90 and 90A in step 156 of FIG.

[0325] like Fig.31 As shown in (A) of FIG. 2 , the display control unit 93 displays a map 2040, a position 2042 on the map based on the position information of the photographing device 10, and a sequence number 2045 on the display screen 2000. The position information and the sequence number of the photographing device 10 are included in the device information.

[0326] like Fig.31 As shown in (B), the display control unit 93 includes: a time series graph 2051 indicating the number of error occurrences based on the error information of the photographing device 10, a time series graph 2053 indicating the temperature change based on the temperature information of the substrate, a time series graph 2055 indicating the temperature change based on the temperature information of the battery, a time series graph 2057 indicating the remaining amount change based on the remaining amount information of the battery, a time series graph 2059 indicating the remaining amount change based on the remaining amount information of the memory, a time series graph 2061 indicating the number of on / off times based on the power on / off information of the photographing device 10, and a time series graph 2063 indicating the number of upload times based on the number of times the image is sent. The error information of the photographing device 10, the temperature information of the substrate, the temperature information of the battery, the remaining amount information of the battery, the remaining amount information of the memory, the power on / off information of the photographing device 10, and the number of times the image is sent are included in the device information.

[0327] Here, in the communication terminal 90 equipped with the application related to transmission and reception developed by our company, even if Fig.31 All the information shown in (A) and (B) is displayed on the display screen 2000. In the communication terminal 90A equipped with the application related to sending and receiving developed by other companies, it is sometimes impossible to send and receive all the device information. Fig.31 Part of the information shown in (A) and (B) will not be displayed on the display screen 2000.

[0328] Fig.32 is a diagram showing an example of a display screen of management information involved in this embodiment, showing Fig.26 Step 140 or Fig. 27 In step 156 , the display control unit 93 of the communication terminal 90 , 90A displays the display screen 2000 on the display 906 .

[0329] The display control unit 93 displays on the display screen 2000: a time series graph 2015 showing the number of members 2010 and changes in the number of members based on the user ID and team ID obtained from the account management DB 5004, a time series graph 2025 showing the number of devices (photographic apparatuses) 2020 and changes in the number of devices based on the virtual device ID obtained from the account management DB 5004, and a time series graph 2035 showing the number of stored images 2030 and changes in the number of images obtained from the image management DB 5002. The user ID and team ID, the virtual device ID, and the number of stored images are included in the management information.

[0330] Here, in the communication terminal 90 equipped with the application program related to transmission and reception developed by our company, even if Fig.32 All the information shown is displayed on the display screen 2000. In the communication terminal 90A equipped with the application program related to sending and receiving developed by other companies, since the team ID is not processed as management information, Fig.32 Among some of the information shown, at least the number of members 2010 and the time series graph 2015 are not displayed on the display screen 2000 .

[0331] Fig.33 is a diagram showing another example of a display screen of management information involved in this embodiment. Fig.26 Step 140 and Fig. 27 In step 156 of FIG. 1 , the display screen 2000 displayed on the display 906 by the display control unit 93 of the communication terminal 90 or 90A is shown.

[0332] The display control unit 93 displays on the display screen 2000 a time series graph 2071 showing changes in the number of image uploads based on the number of image data obtained from the account management DB 5004, a time series graph 2073 showing changes in the number of photographing times based on the number of photographing times obtained from the device information management DB, a time series graph 2075 showing changes in the number of connections based on the number of connections between the application and the photographing device obtained from the device information management DB, and a time series graph 2077 showing changes in the number of logins to the communication terminals 90, 90A as management information, as information on a specific application based on the application ID obtained from the account management DB 5004.

[0333] Fig.33 The information shown is different for each application used in communication terminals 90 and 90A.

[0334] As described above, one embodiment of the present invention involves a photographic device 10 or a communication terminal 90 as an example of a sending source device and a server 50 as an example of a sending destination device, which can be used, for example, by a real estate practitioner who manages or sells real estate, a real estate introduction practitioner who introduces real estate, a construction practitioner who manages structures such as buildings, etc., but the application of the present invention is not limited to this.

[0335] Digital twins, which combine the real world and virtual space, are a new technology that uses the Internet of Things (IoT), artificial intelligence (AI), augmented reality (AR), and virtual reality (VR). In digital twins, data can be collected and analyzed in real time through actual sensors, risks can be assessed, and the causes of failures and errors can be identified. Therefore, digital twins are being used in manufacturing industries such as automobiles and factory production equipment.

[0336] In another embodiment, a mobile object, a production facility in a factory, etc. may have the functional structure of the imaging device described in the above embodiments. Here, the mobile object refers to a vehicle, an airplane, a ship, a mobile robot, etc.

[0337] Fig.34 It is a diagram showing an example of the overall configuration of an image processing system according to a modified example of the embodiment of the present invention.

[0338] Fig.34 The image processing system 1 shown includes a vehicle 70 as an example of a mobile body and a server 50 . The vehicle 70 includes a photographing device 10 as a drive recorder, a control device 70A, and a sensor data storage unit 70B.

[0339] The server 50 constituting the image processing system 1 and the controller 70A of the automobile 70 can communicate with each other via the communication network 100. In addition, the imaging device 10 and the sensor data storage unit 70B can communicate with the control device 70A, respectively.

[0340] That is, the photographing device 10 can communicate with the server 50 via the control device 70A and the communication network 100. In addition, the sensor data storage unit 70B is Fig.13 The modified example of the device information management DB 1003 shown stores and manages data acquired by various sensors included in the vehicle.

[0341] Fig.34 The image processing system 1 shown directly transmits data acquired by various sensors of a vehicle 70 and images captured by a camera 10 as a drive recorder to a server 50 of a vehicle manufacturer.

[0342] The owner of the motor vehicle uploads the data related to the driving of the motor vehicle 70 obtained by various sensors and the images taken by the driving recorder to the server 50 of the motor vehicle manufacturer, so that the motor vehicle manufacturer analyzes the data and images, and signs a service with the motor vehicle manufacturer to provide the owner with various analysis results, such as the owner's driving safety, maintenance or parts replacement notice, etc. In the motor vehicle 70 that has signed the service, a transmission path is set for directly sending the obtained data and the taken images from the wireless communication module of the motor vehicle to the server 50 of the motor vehicle manufacturer.

[0343] Fig.35 FIG. 1 is a diagram showing an example of the overall configuration of an image processing system according to a second modified example of the embodiment of the present invention.

[0344] Fig.35 The illustrated image processing system 1 includes a communication terminal 90 , a vehicle 70 as an example of a mobile body, and a server 50 . The vehicle 70 includes a photographing device 10 as a drive recorder, a control device 70A, and a sensor data storage unit 70B.

[0345] The server 50 and the communication terminal 90 constituting the image processing system 1 can communicate with each other via the communication network 100. In addition, the imaging device 10 and the sensor data storage unit 70B can communicate with the communication terminal 90 via the control device 70A.

[0346] That is, the photographing device 10 can communicate with the server 50 via the control device 70A, the communication terminal 90, and the communication network 100. In addition, the sensor data storage unit 70B is Fig.13 The modified example of the device information management DB 1003 shown stores and manages data acquired by various sensors included in the vehicle.

[0347] Fig.35 The illustrated image processing system 1 transmits data acquired by various sensors of a motor vehicle 70 and images captured by a camera 10 as a drive recorder to a server 50 of an insurance company via a communication terminal 90 .

[0348] The owner of the motor vehicle 70 signs a motor vehicle insurance contract with an insurance company, and promptly notifies the insurance company of the accident when the motor vehicle 70 has an accident. At this time, the owner activates the insurance company's application installed on the communication terminal 90, and the application obtains data related to the motor vehicle 70 at the time of the accident (such as speed, braking timing, steering wheel operation, damaged parts, etc.) and images of the accident taken by the driving recorder from the motor vehicle 70, and uploads the data and images to the insurance company's server 50, so that the insurance company can grasp the accident situation based on the data and images, and obtain the evidence required for negotiations with the other party on the offset of fault, etc.

[0349] In this way, when uploading data from various sensors and captured images in real time, it is possible to distinguish between using a method of directly sending the data and images to be uploaded to server 50 and using a method of sending the data and images to be uploaded via communication terminal 90 only when necessary.

[0350] In another embodiment, for example, the present invention can be applied to a surveillance camera system that monitors banks, retail stores where retailers sell goods, roads managed by municipalities, urban areas, etc. Figure 1 For example, in a surveillance target such as a bank or a retail store, the photographing device 10 is installed at a predetermined surveillance position such as a ceiling or a wall.

[0351] When a road or an urban area is to be monitored, the photographing device 10 is installed on a structure such as a pedestrian bridge or a utility pole. In addition, the server 50 may be owned by an image management company that manages images, or may be owned by a bank, a retail store, or a municipality.

[0352] The communication terminal 90 may be owned by a bank, a retail store, or a municipality, may be owned by a security company that performs security on a surveillance target, or may be owned by both.

[0353] Here, when monitoring the monitored object, real-time (immediacy) is sometimes required in sharing the photographic image. In this case, the photographic device 10 performs image processing with a relatively small load at a level at which the user can check the image, and shares it with the server 50.

[0354] Image processing with a relatively large load, such as object detection processing, is executed in the server 50. Thus, the imaging device 10 can quickly share images on the server 50, and when applied to a monitoring camera system, the captured images can be effectively used.

[0355] Conclusion

[0356] [First Form]

[0357] As described above, the photographing device 10 according to one embodiment of the present invention includes a transmitting and receiving unit 11 that transmits at least one of an image captured by the photographing device 10 and other information other than the image to the server 50 via a first transmission path via the communication terminal 90 or a second transmission path not via the communication terminal 90;

[0358] The transmission and reception unit 11 transmits other information determined by the first transmission path or the second transmission path to the server 50 through the first transmission path or the second transmission path. Here, the server 50 is an example of a transmission destination device, and the communication terminal 90 is an example of an intermediary device.

[0359] Thus, when information is transmitted from the photographing device 10 to the server 50, optimal data collection and data processing can be performed according to the transmission path. That is, the photographing device 10 can transmit other information determined for each transmission path to the server 50.

[0360] The transmitting and receiving unit 11 may transmit other information simultaneously with the image, or may transmit other information at a different time than that of transmitting the image.

[0361] [Second Form]

[0362] In the first embodiment, the photographing device 10 includes an information management unit 17 as an example of a decision mechanism, and determines other information according to the first transmission path or the second transmission path. The decision mechanism can be set in the server 50 or the communication terminal 90. In this case, the transmission and reception unit 11 transmits the other information determined by the server 50 or the communication terminal 90 to the server 50 via the first transmission path or the second transmission path.

[0363] [Third Form]

[0364] In the first form or the second form, the other information is determined by an application program that processes, in the communication terminal 90 , reception of the other information from the imaging device 10 and transmission of the other information to the server 50 .

[0365] Thereby, other information specified in the application installed in the communication terminal 90 can be transmitted from the imaging device 10 to the server 50 via the communication terminal 90 .

[0366] [Fourth Form]

[0367] In the first form or the second form, the photographic device 10 includes an image processing unit 16 that performs image processing on the image, and the sending and receiving unit 11 sends other information corresponding to the image or the processed image on which the image processing is performed to the server 50 through the first sending path or the second sending path, and the other information includes image identification information, and the image identification information indicates whether the image processing is performed or the type of processed image.

[0368] Thus, image recognition information indicating whether image processing has been performed or the type of image processing can be sent from the photographing device 10 to the server 50. Therefore, the device that performs image processing on the image can be any one of the photographing device, the communication terminal, and the server, and flexible image processing can be achieved. When the image to be sent is a photographic image, the sending and receiving unit 11 does not need to send the image recognition information.

[0369] [Fifth Form]

[0370] In the fourth embodiment, the image processing unit 57 performs the first image processing on the photographic image based on the image processing setting information which is information setting the first image processing performed by the photographing device 10 and the second image processing performed by the communication terminal 90 or the server 50 .

[0371] Thus, the image processing can be performed by appropriately sharing the tasks between the photographic device 10 and the communication terminal 90 or the server 50. Specifically, in the first image processing and the second image processing, the processing can be made non-repetitive, or the photographic device 10 can perform the first image processing involving low-load processing, and the communication terminal 90 or the server 50 can perform the second image processing involving high-load processing.

[0372] [Sixth Form]

[0373] In any of the first to fifth aspects, the other information includes the device information of the photographing device 10. Thus, the device information determined for each transmission path can be transmitted from the photographing device 10 to the server 50.

[0374] [Seventh Form]

[0375] A communication terminal 90 involved in one embodiment of the present invention includes a sending and receiving unit 91 as a receiving mechanism, which receives at least one of an image and other information from a photographic device 10. The photographic device 10 sends at least one of an image captured by the photographic device 10 and other information other than the image to a server 50 via a first sending path of the communication terminal 90 or a second sending path not via the communication terminal 90, and the other information is information determined according to the first sending path or the second sending path, respectively. The sending and receiving unit 91 as a sending mechanism sends at least one of the received image, other information and second other information different from the other information to the server 50.

[0376] Thus, while transmitting an image transmitted from the photographing device 10 to the server 50 , other information determined for each transmission path can be transmitted from the photographing device 10 to the server 50 via the communication terminal 90 , or second other information replacing the other information can be transmitted to the server 50 .

[0377] [Eighth Form]

[0378] In the seventh aspect, the communication terminal 90 includes an image processing unit 94 that performs image processing on an image received by the transceiver unit 91 . The transceiver unit 91 as a transmission means transmits a processed image obtained by performing image processing on the received image to the server 50 instead of the received image.

[0379] Thereby, a processed image obtained by performing image processing on the received image can be transmitted to the server 50 instead of the received image.

[0380] [Ninth Form]

[0381] In the eighth aspect, other information includes image identification information, which indicates whether the image received by the sending and receiving unit 91 is a processed image on which image processing has been performed by the photographic device 10, or the type of processed image. The sending and receiving unit 91 receives the image identification information corresponding to the image, and the image processing unit 94 performs image processing based on the image identification information.

[0382] Thus, for example, when the received image is not a processed image, image processing is performed on the received image, and when the received image is a processed image, image processing is not performed on the received image. Thus, the processed image can be reliably sent to the server 50, and repeated execution of the same image processing can be prevented.

[0383] [Tenth Form]

[0384] In the eighth or ninth form, the sending and receiving unit 91 sends second other information corresponding to the image received by the sending and receiving unit 91, or the processed image obtained by performing image processing on the image, to the server 50, and the second other information includes second image identification information, which indicates whether the image sent to the server 50 is the image received by the sending and receiving unit 91 or the processed image, or the type of processed image.

[0385] Thus, on the server 50 side, for example, when the received image is not a processed image, image processing is performed on the received image, and when the received image is a processed image, image processing is not performed on the received image, thereby making it possible to reliably obtain the processed image and prevent repeated execution of the same image processing.

[0386] When the received image is a photographed image and the transmitted image is also the photographed image, the transmitting and receiving unit 91 does not need to transmit the image identification information.

[0387] [Eleventh Form]

[0388] In any one of the eighth to tenth forms, the image processing unit 94 performs at least a second image processing on the image received by the sending and receiving unit 91 based on the image processing setting information, wherein the image processing setting information is information for setting the first image processing performed by the photographic device 10 and the second image processing performed by the communication terminal 90.

[0389] Thus, the image processing can be performed by appropriately sharing the tasks in the photographic device 10 and the communication terminal 90. Specifically, in the first image processing and the second image processing, the processing can be made non-repetitive, or the photographic device 10 can perform the first image processing involving low-load processing, and the communication terminal 90 can perform the second image processing involving high-load processing.

[0390] [Twelfth Form]

[0391] In any of the seventh to eleventh aspects, the other information includes the device information of the photographing device 10. Thus, the device information determined for each transmission path can be transmitted from the photographing device 10 to the server 50 via the communication terminal 90.

[0392] [Thirteenth State]

[0393] In any one of the seventh to twelfth aspects, the second other information includes management information of the imaging device 10 . This allows the management information to be transmitted to the server 50 .

[0394] [Fourteenth Form]

[0395] The server 50 involved in one embodiment of the present invention includes a sending and receiving unit 51, which receives at least one of an image taken by the photographic device 10 and other information other than the image from the photographic device 10 through a first transmission path via a communication terminal 90 or a second transmission path not via the communication terminal 90, and the other information is information determined according to the first transmission path or the second transmission path, respectively.

[0396] Thus, when receiving information from the photographing device 10 , it is possible to collect the most appropriate data and perform data processing according to the transmission path. That is, the server 50 can receive other information determined for each transmission path from the photographing device 10 .

[0397] [Fifteenth Form]

[0398] In the fourteenth aspect, other information includes image identification information indicating whether the image received by the sending and receiving unit 51 is a processed image obtained by performing image processing on the captured image of the photographed subject, or the type of processed image, and the sending and receiving unit 51 receives the image identification information corresponding to the image.

[0399] Thus, the server 50 can understand whether the received image is a processed image or the type of processed image. If the image identification information corresponding to the image is not received, it can be recognized that the received image is a photographed image as is.

[0400] [Sixteenth Form]

[0401] In the fourteenth aspect or the fifteenth aspect, the transmission and reception unit 51 receives the image transmitted from the imaging device 10 through the first transmission path, and receives the second other information different from the other information from the communication terminal 90 .

[0402] Thereby, the image and the second other information can be received from the communication terminal 90 .

[0403] [Seventeenth Form]

[0404] In the sixteenth aspect, the second other information includes second image identification information, which indicates whether the image received by the sending and receiving unit 51 is a processed image obtained by performing image processing by the communication terminal 90 on the image received from the photographic device 10, or the type of processed image, and the sending and receiving unit 51 receives the second image identification information from the communication terminal 90 corresponding to the image.

[0405] Thus, the server 50 can understand whether the received image is a processed image or the type of processed image. If the second image identification information corresponding to the image is not received, it can be recognized that the received image is a photographed image as is.

[0406] [Eighteenth Form]

[0407] In the fifteenth aspect or the seventeenth aspect, the server 50 includes an image processing unit 57 that performs image processing on the image received by the transmission and reception unit based on the image recognition information.

[0408] Thus, on the server 50 side, for example, when the received image is not a processed image, image processing is performed on the received image, and when the received image is a processed image, image processing is not performed on the received image, thereby making it possible to reliably obtain the processed image and prevent repeated execution of the same image processing.

[0409] [Nineteenth Form]

[0410] In any one of the fourteenth to eighteenth forms, the image processing unit 57 performs at least a second image processing on the image received by the sending and receiving unit 51 based on the image processing setting information, wherein the image processing setting information is information setting the first image processing performed by the photographic device 10 or the communication terminal 90 and the second image processing performed by the server 50.

[0411] Thus, the image processing can be performed by appropriately sharing the tasks in the photographing device 10 or the communication terminal 90 and the server 50. Specifically, in the first image processing and the second image processing, the processing can be made non-duplicate, or the first image processing involving low-load processing can be performed in the photographing device 10 or the communication terminal 90, and the second image processing involving high-load processing can be performed in the server 50.

[0412] [Twentieth Form]

[0413] In any one of the fourteenth to nineteenth aspects, the other information includes the device information of the photographing device 10. Thus, it is possible to receive the device information determined for each transmission path from the photographing device 10.

[0414] [Twenty-first Form]

[0415] In the sixteenth aspect or the seventeenth aspect, the second other information includes management information of the imaging device 10. Thus, the management information can be received from the communication terminal 90.

[0416] [Twenty-second Form]

[0417] An embodiment of the present invention involves a sending method performed by a photographic device 10 capable of connecting to a server 50, which performs a sending step for the server 50, and sends at least one of an image captured by the photographic device 10 and other information other than the image through a first sending path via a communication terminal 90 or a second sending path not via the communication terminal 90, wherein the other information is information determined according to the first sending path or the second sending path, respectively.

[0418] [Twenty-third Form]

[0419] The transmission and reception method performed by the communication terminal 90 according to one embodiment of the present invention performs: a receiving step of receiving at least one of an image and other information from the photographing device 10, wherein the photographing device transmits at least one of an image captured by the photographing device 10 and other information other than the image to the server 50 via a first transmission path through the communication terminal 90 or a second transmission path not through the communication terminal 90; and a transmitting step of transmitting at least one of the image, other information, and second other information different from the other information to the server 50. The other information is information determined by the first transmission path or the second transmission path, respectively.

[0420] [State 24]

[0421] An embodiment of the present invention involves a receiving method executed by a server 50 that executes a receiving step, and the execution step receives an image captured by the photographic device 10 and at least one of other information other than the image from the photographic device 10 via a first transmission path of a communication terminal 90, or a second transmission path not via the communication terminal 90, wherein the other information is information determined according to the first transmission path or the second transmission path, respectively.

[0422] [Twenty-fifth Form]

[0423] An embodiment of the present invention involves a program that enables a photographic device 10 that can be connected to a server 50 to execute, and causes a computer to execute a sending step, wherein the sending step sends an image captured by the photographic device 10 and at least one of other information other than the image to the server 50 via a first sending path via a communication terminal 90, or a second sending path not via the communication terminal 90, wherein the other information is information determined according to the first sending path or the second sending path, respectively.

[0424] [Twenty-sixth Form]

[0425] The program executed by the communication terminal 90 according to one embodiment of the present invention causes a computer to execute: a receiving step of receiving at least one of an image and other information from the photographing device 10, wherein the photographing device transmits at least one of an image captured by the photographing device 10 and other information other than the image to the server 50 via a first transmission path through the communication terminal 90 or a second transmission path not through the communication terminal 90; and a transmitting step of transmitting at least one of the image, other information, and second other information different from the other information to the server 50. The other information is information determined by the first transmission path or the second transmission path, respectively.

[0426] [Twenty-seventh Form]

[0427] An embodiment of the present invention involves a program executed by a server 50, which causes a computer to execute a receiving step, which receives an image photographed by the photographic device 10 via a first transmission path of a communication terminal 90, or a second transmission path not via the communication terminal 90, and at least one of other information other than the image, wherein the other information is information determined according to the first transmission path or the second transmission path, respectively.

[0428] [Twenty-eighth Form]

[0429] An image processing system 1 as an example of an information processing system involved in one embodiment of the present invention includes a photographic device 10 and a server 50, the photographic device 10 includes a sending and receiving unit 11, and the sending and receiving unit 11 sends at least one of an image photographed by the photographic device 10 and other information other than the image to the server 50 through a first sending path via a communication terminal 90 or a second sending path not via the communication terminal 90, and the server 50 includes a sending and receiving unit 51, and the sending and receiving unit 51 receives at least one of the image and other information from the photographic device 10 through the first sending path or the second sending path, and the other information is information determined according to the first sending path or the second sending path, respectively.

[0430] [Twenty-ninth Form]

[0431] In the twenty-eighth aspect, the image processing system 1 further includes a communication terminal 90, which includes a sending and receiving unit 91 as a receiving mechanism, which receives images and other information from the photographic device 10, and the sending and receiving unit 91 as a sending mechanism sends the image, other information, and at least one of the second other information different from the other information to the server 50.

[0432] [Thirtieth Form]

[0433] A computer program product involved in one embodiment of the present invention enables a photographic device that can be connected to a destination device to execute a sending step, wherein the sending step sends at least one of an image captured by the photographic device and other information other than the image to the destination device through a first sending path via an intermediate device, or a second sending path not via the intermediate device; the other information is information determined according to the first sending path or the second sending path, respectively.

[0434] [Thirty-first Form]

[0435] A computer program product according to one embodiment of the present invention causes an intermediary device to execute, so that a computer executes:

[0436] a receiving step of receiving an image and other information from an imaging device, wherein the imaging device transmits at least one of the image captured by the imaging device and other information other than the image to a destination device via a first transmission path passing through an intermediate device or a second transmission path not passing through the intermediate device; and

[0437] a sending step of sending at least one of the other information and second other information different from the other information to the sending destination device,

[0438] The other information is information determined according to the first transmission path or the second transmission path.

[0439] [Thirty-second Form]

[0440] A computer program product involved in one embodiment of the present invention causes an information processing device to execute, so that the computer executes a receiving step, wherein the receiving step receives at least one of an image captured by the photographic device and other information other than the image from the photographic device through a first transmission path via an intermediate device or a second transmission path not via the intermediate device; the other information is information determined according to the first transmission path or the second transmission path, respectively.

[0441] [Thirty-third Form]

[0442] A computer-readable storage medium according to one embodiment of the present invention stores a program for a computer to execute the following steps: causing a photographic device 10 connectable to a server 50 to execute a sending step, wherein the sending step sends at least one of an image captured by the photographic device and other information other than the image to a destination device via a first sending path via an intermediary device or a second sending path not via the intermediary device; the other information is information determined according to the first sending path or the second sending path, respectively.

[0443] [Thirty-fourth Form]

[0444] A computer-readable storage medium according to an embodiment of the present invention stores a computer-executable program, which causes an intermediary device to execute:

[0445] a receiving step of receiving an image and other information from an imaging device, wherein the imaging device transmits at least one of the image captured by the imaging device and other information other than the image to a destination device via a first transmission path passing through an intermediate device or a second transmission path not passing through the intermediate device; and

[0446] a sending step of sending at least one of the other information and second other information different from the other information to the sending destination device,

[0447] The other information is information determined according to the first transmission path or the second transmission path.

[0448] [Thirty-fifth Form]

[0449] A computer-readable storage medium according to one embodiment of the present invention stores a program for a computer to execute the following steps: causing an information processing device to execute a receiving step, wherein the receiving step receives at least one of an image captured by the photographic device and other information other than the image from the photographic device via a first sending path via an intermediate device or a second sending path not via the intermediate device; the other information is information determined according to the first sending path or the second sending path, respectively.

[0450] [Thirty-sixth Form]

[0451] An imaging device 10 according to one embodiment of the present invention includes:

[0452] The camera unit 101 captures images and obtains the captured images; and

[0453] The sending and receiving unit 11 sends other information different from the captured image determined by a first sending path through a communication terminal 90 (intermediary device) and a second sending path not through the communication terminal 90 (intermediary device) to a server 50 (sending destination device).

[0454] [Thirty-seventh Form]

[0455] The thirty-sixth aspect further includes a data management unit 17 that determines the other information to be transmitted according to the first transmission path and the second transmission path that does not pass through the communication terminal 90 (intermediary device).

[0456] [Thirty-eighth Form]

[0457] In the thirty-sixth form or the thirty-seventh form, the data management unit 17 determines the other information based on the application (program) installed in the above-mentioned communication terminal 90 (intermediary device), and the application (program) processes the reception of the other information from the photographic device 10 and the sending of the other information to the server 50 (sending destination device).

[0458] [Thirty-ninth Form]

[0459] In the thirty-sixth or thirty-seventh form,

[0460] further comprising an image processing unit 16 for performing a first image processing on the captured image;

[0461] The transmission and reception unit 11 transmits the other information in correspondence with the captured image or the processed image on which the image processing is performed to the server 50 (sending destination device) through one of the first sending path and the second sending path that does not pass through the communication terminal 90 (intermediary device);

[0462] The other information includes image identification information indicating whether the image processing has been executed or the type of the processed image.

[0463] [Fortieth Form]

[0464] In the thirty-ninth aspect, the image processing unit 16 performs the first image processing on the photographic image according to the image processing setting information, and the image processing setting information is information setting the first image processing performed and the second image processing performed by the communication terminal 90 (intermediary device) or the server 50 (sending destination device).

[0465] [Form 41]

[0466] In any one of the 36th to 40th aspects, the other information includes device information of the photographing device 10 .

[0467] [Form 42]

[0468] An information processing system 1 according to one embodiment of the present invention comprises:

[0469] The imaging device 10 according to any one of the thirty-sixth aspect to the forty-first aspect; and

[0470] Server 50 (sending destination device),

[0471] The server 50 (destination device) includes a transceiver 51 that receives the other information from the imaging device 10 via one of a first transmission path and a second transmission path that does not pass through the communication terminal 90 (intermediary device).

[0472] [Form 43]

[0473] The 42nd aspect further includes the above-mentioned communication terminal 90 (intermediary device), which is provided with a sending and receiving unit 91, which receives the other information from the photographic device and sends the other information and second other information different from the other information to the sending destination device.

[0474] ●Supplement●

[0475] The functions involved in the above-described embodiments can be implemented by one or more processing circuits. Here, the "processing circuit" in this embodiment refers to a processor that is programmed by software to perform various functions, such as a processor implemented by an electronic circuit, and an ASIC (application-specific integrated circuit), DSP (digital signal processor), FPGA (field programmable gate array), SOC (system on chip), GPU (graphics processing unit) and existing circuit modules designed to perform the functions described above.

[0476] The present invention is not limited to the above-mentioned embodiments, and the addition, change or deletion of other embodiments may be changed within the scope that can be conceived by those skilled in the art. No matter in which form, as long as the function and effect of the present invention are achieved, it is included in the scope of the present invention.

Claims

1. A photographing device that can be connected to a destination device, characterized in that : A transmission mechanism is included, wherein the transmission mechanism transmits at least one of the image captured by the photographing device and other information other than the image to the destination device through a first transmission path passing through an intermediate device or a second transmission path not passing through the intermediate device; The transmission unit transmits the other information determined according to the first transmission path or the second transmission path to the transmission destination device through the first transmission path or the second transmission path.

2. The photographing device according to claim 1, characterized in that: A decision mechanism is included to determine the other information according to the first transmission path or the second transmission path.

3. The photographing device according to claim 1, characterized in that: The other information is determined by an application, and the application processes, in the intermediary device, receiving the other information from the imaging device and transmitting the other information to the destination device.

4. The photographing device according to claim 1 or 2, characterized in that: including an image processing mechanism for performing image processing on the image; The transmission means transmits the other information to the transmission destination device via the first transmission path or the second transmission path in association with the image or the processed image on which the image processing is performed; The other information includes image identification information indicating whether the image processing has been executed or the type of the processed image.

5. The photographing device according to claim 4, characterized in that: The image processing mechanism performs a first image processing on the photographic image based on image processing setting information, wherein the image processing setting information is information setting a first image processing to be performed by the photographic device and a second image processing to be performed by the intermediate device or the destination device.

6. The photographing device according to claim 1, characterized in that: The other information includes equipment information of the photographic device.

7. An intermediary device, characterized in that: The device comprises a receiving mechanism for receiving an image and other information from a photographing device, wherein the photographing device transmits at least one of an image captured by the photographing device and other information other than the image to a destination device via a first transmission path passing through the intermediate device or a second transmission path not passing through the intermediate device; The other information is information determined according to the first transmission path or the second transmission path respectively; The device further includes a transmission unit that transmits at least one of the image, the other information, and second other information different from the other information to the destination device.

8. The mediation device according to claim 7, characterized in that: comprising an image processing unit that performs image processing on the image received by the receiving unit, The transmission means transmits, to the transmission destination device, a processed image obtained by performing the image processing on the image, instead of the image.

9. The mediation device according to claim 8, characterized in that: The other information includes image identification information indicating whether the image received by the receiving unit is a processed image on which image processing has been performed by the photographing device, or the type of the processed image; The receiving mechanism receives the image recognition information corresponding to the image; The image processing mechanism performs the image processing based on the image recognition information.

10. The mediation device according to claim 8 or 9, characterized in that: the transmitting means transmitting, to the transmission destination device, the second other information corresponding to the image received by the receiving means or a processed image obtained by performing the image processing on the image; The second other information includes second image identification information indicating whether the image transmitted to the transmission destination device is the image received by the reception unit or the processed image, or the type of the processed image.

11. The mediation device according to claim 8, characterized in that: The image processing mechanism performs at least the second image processing on the image received by the receiving mechanism according to image processing setting information, wherein the image processing setting information is information setting the first image processing performed by the photographic device and the second image processing performed by the intermediary device.

12. The mediation device according to claim 7, characterized in that: The other information includes equipment information of the photographic device.

13. The mediation device according to claim 7, characterized in that: The second other information includes management information of the photographing device.

14. An information processing device, characterized in that: A receiving mechanism is included, wherein the receiving mechanism receives at least one of the image captured by the photographing device and other information other than the image through a first transmission path passing through an intermediary device or a second transmission path not passing through the intermediary device; The other information is information determined according to the first transmission path or the second transmission path.

15. The information processing device according to claim 14, characterized in that: The other information includes image identification information indicating whether the image received by the receiving unit is a processed image obtained by performing image processing on a photographic image of a subject, or the type of the processed image; The receiving mechanism receives the image recognition information corresponding to the image.

16. The information processing device according to claim 14, characterized in that: The receiving mechanism receives the image transmitted from the photographing device on the first transmission path, and receives second other information different from the other information from the mediation device.

17. The information processing device according to claim 16, characterized in that: The second other information includes second image identification information indicating whether the image received by the receiving mechanism is a processed image obtained by the intermediary device performing image processing on the image received from the photographing device, or the type of the processed image; The receiving unit receives the second image identification information corresponding to the image from the intermediary device.

18. The information processing device according to claim 15 or 17, characterized in that: An image processing unit is included, and the image processing unit performs image processing on the image received by the receiving unit according to the image recognition information.

19. The information processing device according to claim 18, characterized in that: The image processing mechanism performs at least the second image processing on the image received by the receiving mechanism according to image processing setting information, wherein the image processing setting information is information setting the first image processing performed by the photographic device or the intermediary device and the second image processing performed by the information processing device.

20. The information processing device according to claim 14, characterized in that: The other information includes equipment information of the photographing device.

21. The information processing device according to claim 16 or 17, characterized in that: The second other information includes management information of the photographing device.

22. A transmission method executed by a photographing device that can be connected to a transmission destination device, characterized in that: A sending step is performed on the destination device to send at least one of the image captured by the photographing device and other information other than the image through a first sending path via an intermediary device or a second sending path not via the intermediary device; The other information is information determined according to the first transmission path or the second transmission path.

23. A sending and receiving method executed by an intermediary device, characterized in that: The sending and receiving method performs: A receiving step of receiving at least one of an image and other information from a photographing device, wherein the photographing device transmits at least one of an image captured by the photographing device and other information other than the image to a destination device via a first transmission path passing through an intermediary device or a second transmission path not passing through the intermediary device; as well as a sending step of sending at least one of the image, the other information, and second other information different from the other information to the sending destination device, The other information is information determined according to the first transmission path or the second transmission path.

24. A receiving method executed by an information processing device, characterized in that: executing a receiving step, wherein the executing step receives at least one of an image captured by the photographic device and other information other than the image from the photographic device through a first transmission path passing through an intermediary device or a second transmission path not passing through the intermediary device; The other information is information determined according to the first transmission path or the second transmission path.

25. A computer program product, which is executed by a photographing device that can be connected to a transmission destination device, characterized in that: causing the computer to execute a sending step of sending at least one of an image captured by the photographing device and other information other than the image to a destination device via a first sending path passing through an intermediary device or a second sending path not passing through the intermediary device; The other information is information determined according to the first transmission path or the second transmission path.

26. A computer program product, which enables an intermediary device to execute, characterized in that: Causes the computer to execute: A receiving step, wherein the receiving step receives an image and other information from a photographing device, wherein the photographing device transmits at least one of the image captured by the photographing device and other information other than the image to a destination device via a first transmission path passing through an intermediate device or a second transmission path not passing through the intermediate device; as well as a sending step of sending at least one of the other information and second other information different from the other information to the sending destination device, The other information is information determined according to the first transmission path or the second transmission path.

27. A computer program product for executing on an information processing device, characterized in that: causing the computer to execute a receiving step of receiving at least one of an image captured by the photographic device and other information other than the image from the photographic device via a first transmission path passing through an intermediary device or a second transmission path not passing through the intermediary device; The other information is information determined according to the first transmission path or the second transmission path.

28. An information processing system comprising a photographing device and a destination device, characterized in that: The photographic device includes a transmission mechanism that transmits at least one of an image captured by the photographic device and other information other than the image to the destination device via a first transmission path that passes through an intermediary device or a second transmission path that does not pass through the intermediary device; The transmission destination device includes a receiving mechanism that receives at least one of the image and the other information from the photographing device via the first transmission path or the second transmission path; The other information is information determined according to the first transmission path or the second transmission path.

29. The information processing system according to claim 28, characterized in that: Further comprising a mediating device; The intermediary device comprises: A receiving mechanism receives the image and other information from the photographic device; and The transmission means transmits at least one of the image, the other information, and second other information different from the other information to the transmission destination device.

Images