Information processing system, and information processing method

Abstract

To execute detection of an abnormality occurring in a photographing device, by using an appropriate device and an appropriate method.SOLUTION: An information processing system has a photographing device that performs photographing to acquire a photographed image, and an information processing apparatus that can communicate with the photographing device. The information processing system has storage means that stores setting information associated with a defect in the photographed image and a notification destination of the defect, detection means that detects the defect in the photographed image, and notification means that notifies the notification destination of the detection of the defect on the basis of the setting information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system and an information processing method. [Background technology]

[0002] Patent Document 1 describes a surveillance camera system in which multiple imaging devices and a system management terminal that performs system management control are connected via a network, and the system is equipped with an abnormality detection means that uses an image difference identification means that identifies differences in images of the same range within the shooting range captured by the multiple imaging devices to detect abnormalities occurring in the images of the imaging devices, and an alarm means that issues an alarm to the system management terminal when an abnormality is detected. In Patent Document 1, abnormality detection and alarms are only performed on the system management terminal, but this method is only effective for surveillance cameras in which the imaging devices are fixed and unmanned. For example, if the imaging device is moved and captured by a person, abnormalities must be detected and alarms issued at the shooting site. Furthermore, some abnormality detection items can only be processed within the imaging device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-89031 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology proposed in the aforementioned Patent Document 1 is configured to detect anomalies individually for each surveillance camera and issue a warning to the system administrator. Therefore, when a failure occurs in the network section of the camera, it is impossible to issue a warning from the camera.

[0005] An object of the present disclosure is to detect an abnormality occurring in an imaging device at an appropriate location and by an appropriate method. [Means for solving the problem]

[0006] The present disclosure relates to an information processing system having an imaging device that acquires an image by taking a photograph, and an information processing device that can communicate with the imaging device, the information processing system having a storage means that stores setting information that associates defects in the captured image with a notification destination of the defect, a detection means that detects defects in the captured image, and a notification means that notifies the notification destination that the defect has been detected based on the setting information. [Effects of the Invention]

[0007] According to the present disclosure, detection of an abnormality occurring in an imaging device can be performed at an appropriate location and by an appropriate method. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an image processing system according to an embodiment of the present invention. [Figure 2] (a) is a hemispherical image (front) captured by a camera, (b) is a hemispherical image (back) captured by a camera, and (c) is an image represented by equirectangular projection. [Figure 3] (a) is a conceptual diagram showing how a sphere is covered with an equirectangular projection image, and (b) is a diagram showing a spherical image. [Figure 4] FIG. 10 is a diagram showing the positions of a virtual camera and a predetermined area when the celestial sphere image is a three-dimensional sphere. [Figure 5] 10 is a diagram showing the relationship between predetermined area information and an image of a predetermined area T. FIG. [Figure 6] FIG. 1 is a diagram illustrating an example of a hardware configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 7] FIG. 2 is a diagram illustrating an example of the hardware configuration of a server and a communication terminal according to the present embodiment. [Figure 8] FIG. 1 is a diagram illustrating an example of a functional configuration of an image processing system according to an embodiment of the present invention. [Figure 9]FIG. 10 is a conceptual diagram illustrating an example of a Twin management table according to the present embodiment. [Figure 10] FIG. 10 is a sequence diagram showing an example of a Twin registration process according to the present embodiment. [Figure 11] FIG. 1 is a diagram illustrating a projection relationship in an imaging device using a fisheye lens. [Figure 12] FIG. 1 is a diagram illustrating a data structure of image data in a spherical image format. [Figure 13] 1 is a flowchart for generating a spherical image by stitching together images taken with two fisheye lenses. [Figure 14] FIG. 10 is a diagram illustrating a conversion table for converting a fisheye image into a spherical image. [Figure 15] 10A and 10B are diagrams illustrating mapping of two partial images captured by two fisheye lenses onto a spherical coordinate system during position detection processing. [Figure 16] FIG. 10 is a diagram illustrating a connection position detection process. [Figure 17] 10A and 10B are diagrams illustrating mapping of partial images captured by a fisheye lens onto a spherical coordinate system during image synthesis processing. [Figure 18] FIG. 4 is a sequence diagram showing a first abnormality detection process according to the present embodiment. [Figure 19] FIG. 4 is a sequence diagram showing a first abnormality detection process according to the present embodiment. [Figure 20] FIG. 10 is a diagram illustrating an example of an abnormality detection setting file according to the present embodiment. [Figure 21] FIG. 10 is a diagram showing an example of an abnormality detection setting screen. [Figure 22] FIG. 2 is a diagram showing an example of an abnormality detection setting screen 2. [Figure 23] FIG. 10 is a diagram showing an example of an abnormality management file according to the embodiment. [Figure 24] FIG. 10 is a diagram showing an example of an error code for each abnormality detection item. [Figure 25] FIG. 2 is a diagram illustrating a recording format of detection data. [Figure 26]FIG. 10 is a sequence diagram showing a second abnormality detection process according to the present embodiment. [Figure 27] FIG. 10 is a sequence diagram showing a second abnormality detection process according to the present embodiment. [Figure 28] FIG. 10 is a sequence diagram showing a third abnormality detection process according to the present embodiment. [Figure 29] FIG. 10 is a sequence diagram showing a third abnormality detection process according to the present embodiment. [Figure 30] FIG. 10 is a sequence diagram showing a fourth abnormality detection process according to the present embodiment. [Figure 31] FIG. 10 is a sequence diagram showing a fourth abnormality detection process according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated explanations will be omitted.

[0010] <Image processing system overview> Fig. 1 is a diagram showing an example of the overall configuration of an image processing system according to an embodiment of the present invention. The image processing system 1 shown in Fig. 1 is an example of an information processing system, and is a system that performs image processing on captured images to allow viewers to view the interior space of structures such as real estate and architectural properties, and structures such as construction and civil engineering structures online.

[0011] As shown in FIG. 1 , the image processing system 1 includes an image capturing device 10, a server 50, and a communication terminal 90, and is connectable to an external server 70. The server 50 and the communication terminal 90 constituting the image processing system 1 can communicate with each other via a communication network 100. The server 50 can communicate with the external server 70 via the communication network 100. The image capturing device 10 can communicate with the communication terminal 90 via short-range wireless communication such as Wi-Fi, but may also communicate with the server 50 and the communication terminal 90 via the communication network 100. The communication network 100 is constructed using the Internet, a mobile communication network, a LAN (Local Area Network), or the like. Note that the communication network 100 may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0012] The server 50 is an example of an information processing device or a destination device, and is a server computer that stores photographed data of the interior space of a structure, such as a real estate property or building, or a construction or civil engineering structure, which is a predetermined base. The server 50, for example, acquires photographed images captured by the image capture device 10 and generates tour images to provide a user with a virtual tour using the acquired photographed images. Here, a virtual tour is content that allows a user to view a real estate property or the like as if they were actually on-site. The tour images are generated using multiple photographed images captured by the image capture device 10 and are images for viewing that allow the user to virtually move around the base depicted in the photographed images through user operation. Such virtual tours are not limited to the interior space of a structure at a real estate property or a construction site, but can also be suitably implemented in the exterior space of a structure, such as a tourist destination or a theme park. In other words, the interior space of a structure can also be considered as within a predetermined area. When implementing a virtual tour in the exterior space of a structure, the present embodiment can be implemented by replacing the map showing the interior space of a structure, such as a real estate property or building, described below, with, for example, a tourist destination map introducing tourist destinations or an area map of a theme park.

[0013] The server 50 may be configured by one server computer or by multiple server computers. Although the server 50 will be described as a server computer existing in a cloud environment, it may also be a server existing in an on-premise environment.

[0014] The image capturing device 10 is an example of an information processing device or a transmission source device, and is a special digital camera (omnidirectional image capturing device) that can capture an image of a celestial sphere (360° in both the circumferential and vertical directions) by capturing an image of a space where a structure such as a real estate property exists at an image capturing base in all directions. The following description will be given mainly using an example of a structure such as a real estate property, but as mentioned above, the image capturing device 10 may also be the interior space of a structure such as a building property, or a structure for construction or civil engineering.

[0015] Note that a celestial sphere image refers to an image with a so-called solid angle of 4πsr [sr: steradian]. For convenience, in this specification, an image in which a part of a celestial sphere image is missing is also referred to as a celestial sphere image. For example, this is an image in which a part of the area directly above or directly below the celestial sphere imaging device is missing, a part of the area vertically above or vertically below the celestial sphere image is missing, or a part of a predetermined area of ​​the celestial sphere image is missing.

[0016] For example, there may be usage scenarios in which the areas directly above or below a subject in a spherical image are not looked at very carefully when viewing the image. In such cases, it is conceivable to design the image sensor and optical system so that those areas are not captured in the first place, or not display the image, or to display a logo or the like in those areas, or to not display the spherical image itself.

[0017] The image capturing device 10 is used, for example, by a real estate agent who manages or sells real estate properties. The image capturing device 10 may be a wide-angle camera or a stereo camera capable of capturing wide-angle images with a field of view greater than a predetermined value. A wide-angle image is generally an image captured using a wide-angle lens (wide-angle image), and is an image captured using a lens that can capture a wider range than what the human eye can perceive. In other words, the image capturing device 10 is a capturing means capable of capturing images (spherical images, wide-angle images) captured using a lens with a focal length shorter than a predetermined value. A wide-angle image generally refers to an image captured using a lens with a focal length of 35 mm or less in 35 mm film equivalent. The image capturing device 10 may also have a panoramic shooting function in its shooting function and may be capable of capturing panoramic images.

[0018] The communication terminal 90 is an example of an information processing device or a source device, and is a computer such as a tablet terminal. The communication terminal 90 is used, for example, by the same real estate agent as the image capture device 10. A dedicated application for issuing image capture instructions to the image capture device 10 and viewing images provided by the server 50 is installed on the communication terminal 90. Alternatively, the communication terminal 90 may be configured to issue image capture instructions and view images by accessing a dedicated website using a web browser without using a dedicated application. Alternatively, the image capture instructions and image viewing may be performed by different communication terminals 90.

[0019] The communication terminal 90 is not limited to a tablet terminal, but may be, for example, a PC, a smartphone, a wearable terminal, an HMD (head mounted display), or an IWB (Interactive White Board: an electronic whiteboard with a blackboard function that allows mutual communication), etc.

[0020] <Outline of the imaging device> 2 to 5, an overview of the image capturing device 10 constituting the image processing system 1 will be described. The image capturing device 10 captures a spherical image. Spherical images come in a variety of forms, but are often generated using the equirectangular (equirectangular projection) projection method described below. Images generated using this equirectangular projection have the advantages of having a rectangular outer shape, making it easy and efficient to store image data, and of appearing relatively natural because there is little distortion near the equator and vertical straight lines are not distorted.

[0021] <Outline of spherical image> Next, an overview of a spherical image will be described. First, an overview of the process of generating a spherical image from an image captured by the image capturing device 10 will be described using Fig. 2 and Fig. 3. Fig. 2(a) is a diagram showing a hemispherical image (front side) captured by the image capturing device 10, Fig. 2(b) is a diagram showing a hemispherical image (rear side) captured by the image capturing device 10, and Fig. 2(c) is a diagram showing an image expressed by equirectangular projection (hereinafter referred to as "equirectangular projection image"). Fig. 3(a) is a conceptual diagram showing a state in which a sphere is covered with an equirectangular projection image, and Fig. 3(b) is a diagram showing a spherical image.

[0022] The photographing device 10 is provided with image sensors on both the front side (front side) and the back side (rear side). These image sensors (image sensors) are used in conjunction with optical components such as lenses that can capture hemispherical images (angle of view of 180° or more). The photographing device 10 can obtain two hemispherical images by capturing images of subjects around the user using the two image sensors.

[0023] 2(a) and 2(b), the images captured by the imaging element of the image capturing device 10 are curved hemispherical images (front and rear). The image capturing device 10 then combines the hemispherical image (front) with a hemispherical image (rear) flipped 180 degrees to create an equirectangular projection image EC as shown in FIG. 2(c).

[0024] Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), the imaging device 10 applies an equirectangular projection image EC to cover the spherical surface as shown in FIG. 3(a), thereby creating a celestial sphere image (celestial sphere panoramic image) CE as shown in FIG. 3(b). In this way, the celestial sphere image CE is expressed as an image in which the equirectangular projection image EC faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. The celestial sphere image CE may be a still image or a video. Furthermore, the conversion method is not limited to OpenGL ES, and any method capable of converting a hemispherical image into an equirectangular projection may be used, for example, a CPU operation or OpenCL operation.

[0025] As described above, the spherical image CE is an image pasted to cover the spherical surface, which gives a sense of incongruity to people. Therefore, the imaging device 10 can display a predetermined region T (hereinafter referred to as a "predetermined region image") that is a part of the spherical image CE as a planar image with little curvature, thereby enabling a display that does not give a sense of incongruity to people. This will be described with reference to FIGS. 4 and 5.

[0026] FIG. 4 is a diagram showing the positions of the virtual camera and the predetermined region when the celestial sphere image is a three-dimensional sphere. The virtual camera IC corresponds to the viewpoint of a user viewing the celestial sphere image CE displayed as a three-dimensional sphere. FIG. 4 represents the celestial sphere image CE as a three-dimensional sphere CS. If the celestial sphere image CE generated in this manner is a three-dimensional sphere CS, as shown in FIG. 4, the virtual camera IC is located inside the celestial sphere image CE. The predetermined region T in the celestial sphere image CE is the shooting region of the virtual camera IC and is specified by predetermined region information that indicates the shooting direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the celestial sphere image CE. Zooming of the predetermined region T can also be expressed by moving the virtual camera IC closer to or farther away from the celestial sphere image CE. The predetermined region image Q is an image of the predetermined region T in the celestial sphere image CE. Therefore, the predetermined region T can be specified by the angle of view α and the distance f from the virtual camera IC to the celestial sphere image CE.

[0027] The predetermined area image Q is then displayed on a predetermined display as an image of the shooting area of ​​the virtual camera IC. The following description will be given using the shooting direction (ea, aa) and angle of view (α) of the virtual camera IC. Note that the predetermined area T may be represented by the shooting area (X, Y, Z) of the virtual camera IC, which is the predetermined area T, instead of the angle of view α and the distance f.

[0028] Next, the relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 5. FIG. 5 is a diagram showing the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 5, "ea" represents the elevation angle, "aa" represents the azimuth angle, and "α" represents the angle of view (Angle). That is, the attitude of the virtual camera IC is changed so that the gaze point of the virtual camera IC, indicated by the shooting direction (ea, aa), becomes the center point CP(x, y) of the predetermined area T, which is the shooting area of ​​the virtual camera IC. As shown in FIG. 6, when the diagonal angle of view of the predetermined area T, represented by the angle of view α of the virtual camera IC, is α, the center point CP(x, y) becomes the parameter ((x, y)) of the predetermined area information. The predetermined area image Q is an image of the predetermined area T in the celestial sphere image CE. f represents the distance from the virtual camera IC to the center point CP(x, y). L represents the distance between any vertex of the predetermined area T and the center point CP(x, y) (2L is the diagonal). In FIG. 5, the trigonometric function shown in the following (Equation 1) generally holds. (L / f)=tan(α / 2) (Equation 1) <<Hardware configuration>> Next, the hardware configuration of each device or terminal constituting the image processing system according to the embodiment will be described with reference to Figures 6 and 7. Note that components may be added or deleted from the hardware configurations shown in Figures 6 and 7 as needed.

[0029] <Hardware configuration of the imaging device> First, the hardware configuration of the image capturing device 10 will be described using Figure 6. Figure 6 is a diagram showing an example of the hardware configuration of an image capturing device. In the following, the image capturing device 10 is assumed to be an omnidirectional (omnidirectional) image capturing device using two image capturing elements, but the number of image capturing elements may be two or more. Furthermore, the image capturing device does not necessarily have to be a device dedicated to omnidirectional image capturing; an omnidirectional image capturing unit may be attached to a regular digital camera, smartphone, or the like to have substantially the same functions as the image capturing device 10.

[0030] 6, the photographing device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F (Interface) 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, and a network I / F 121. The photographing device 10 also includes an ISP (Image Signal Processor) that processes video signals (not shown).

[0031] Of these, the imaging unit 101 includes optical systems (wide-angle lenses or so-called fisheye lenses) 102a and 102b (hereinafter referred to as lenses 102 when there is no need to distinguish between them) each having a field of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b corresponding to each lens. Here, the combination of one optical system and one imaging element is called an imaging optical system, and the imaging device 10 can be configured by arranging two imaging optical systems facing each other. Note that the imaging device 10 can also be configured using two or more imaging optical systems. The imaging elements 103a and 103b include image sensors such as CMOS (Complementary Metal Oxide Semiconductor) sensors or CCD (Charge Coupled Device) sensors that convert optical images captured by the lenses 102a and 102b into electrical image data signals and output them, a timing generation circuit that generates horizontal and vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands or parameters required for the operation of the imaging elements are set.

[0032] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. Furthermore, the bus 110 is also connected to a ROM 112, an SRAM 113, a DRAM 114, an operation unit 115, an input / output I / F 116, a short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120, a network I / F 121, an LCD 122, and the like.

[0033] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each image data, and then synthesizes the image data to create equirectangular projection image data as shown in Figure 2(c).

[0034] The imaging control unit 105 generally sets commands and the like in the registers of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the registers of the imaging elements 103a and 103b and send it to the CPU 111.

[0035] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button on the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function for displaying moving images on a display (for example, the LCD 122 or the display of an external terminal such as a smartphone that communicates with the imaging device 10 by using the short-range communication circuit 117). In this case, the image data is output continuously from the imaging elements 103a and 103b at a predetermined frame rate (frames / second).

[0036] As will be described later, the imaging control unit 105 also functions as a synchronization control means that cooperates with the CPU 111 to synchronize the output timing of image data from the imaging elements 103a and 103b. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 receives the sound data output from the microphone 108 via an I / F bus and performs predetermined processing on the sound data.

[0037] The CPU 111 controls the overall operation of the image capturing device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and DRAM 114 are work memories that store programs executed by the CPU 111, data in the middle of processing, etc. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and data of processed equirectangular projection images.

[0038] The operation unit 115 is a general term for various operation buttons, a power switch, a shutter button, a touch panel that combines display and operation functions, etc. The user operates the operation unit 115 to input various shooting modes, shooting conditions, etc.

[0039] The input / output I / F 116 is a general term for an interface circuit (such as an SD memory card I / F or USB I / F) with an external medium such as an SD memory card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as needed.

[0040] The short-range communication circuit 117 communicates with an external terminal (device) by short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi via an antenna 117a provided in the image capturing device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0041] The electronic compass 118 calculates the orientation of the image capture device 10 from the Earth's magnetism and outputs orientation information. This orientation information is an example of related information (metadata) conforming to the Exif standard and is used for image processing such as image correction of captured images. The related information also includes data such as the capture date and time of the image and the data size of the image data. The gyro sensor 119 detects angle changes (roll angle, pitch angle, and yaw angle) associated with the movement of the image capture device 10. The angle changes are an example of related information (metadata) conforming to the Exif standard and are used for image processing such as image correction of captured images. The acceleration sensor 120 detects acceleration in three axial directions. The image capture device 10 calculates the attitude (angle with respect to the direction of gravity) of its own device (the image capture device 10) based on the acceleration detected by the acceleration sensor 120. The inclusion of the acceleration sensor 120 in the image capture device 10 improves the accuracy of image correction. The network I / F 121 is an interface for data communication using a communication network 100 such as the Internet via a router or the like. The LCD (Liquid Crystal Display) 122 is a device that can display the photographing settings of the photographing device 10 and images and videos photographed by the photographing device 10.

[0042] <Server hardware configuration> Next, the hardware configuration of server 50 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the hardware configuration of a server. Each piece of hardware configuration in server 50 is designated by a reference number in the 500 series. Server 50 is constructed by a computer, and as shown in Fig. 7, includes a CPU 501, a ROM 502, a RAM (Random Access Memory) 503, a HD (Hard Disk) 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0043] Of these, the CPU 501 controls the overall operation of the server 50. The ROM 502 stores programs used to drive the CPU 501, such as an IPL (Initial Program Loader). The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The display 506 may be a touch panel display equipped with input means. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB memory. The network I / F 509 is an interface for data communication using the communication network 100. The bus line 510 is an address bus, a data bus, or the like, for electrically connecting the components, such as the CPU 501, shown in FIG. 7.

[0044] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting or executing various instructions, selecting a processing target, moving a cursor, etc. The input means may be not only the keyboard 511 and the pointing device 512, but also a touch panel, a voice input device, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513, which is an example of a removable recording medium. The removable recording medium is not limited to a DVD-RW, but may also be a DVD-R or a Blu-ray (registered trademark) Disc, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory.

[0045] <Hardware configuration of communication terminal> Fig. 7 also shows an example of the hardware configuration of the communication terminal 90. Each piece of hardware configuration of the communication terminal 90 is indicated by a reference number in the 900 series in parentheses. The communication terminal 90 is constructed by a computer, and as shown in Fig. 7, has the same configuration as the server 50, so a description of each piece of hardware configuration will be omitted. In addition to the same configuration as the server 50, the communication terminal 90 also has a short-range communication circuit 917 and an antenna 917a for the short-range communication circuit 917. The short-range communication circuit 917 is a communication circuit such as NFC, Bluetooth, or Wi-Fi.

[0046] Each of the above programs may be recorded as an installable or executable file on a computer-readable recording medium and distributed. Examples of recording media include CD-Rs (Compact Disc Recordables), DVDs (Digital Versatile Disks), Blu-ray Discs, SD cards, and USB memory sticks. The recording media may also be provided domestically or internationally as a program product. For example, the server 50 executes the program according to the present invention to realize the image processing method according to the present invention.

[0047] <<Function configuration>> Next, the functional configuration of the image processing system according to the embodiment will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of the functional configuration of the image processing system. Fig. 8 shows devices or terminals shown in Fig. 1 that are related to the processing or operation described below.

[0048] <Functional configuration of the imaging device> First, the functional configuration of the photographing device 10 will be described with reference to Fig. 8. The photographing device 10 has a transmitting / receiving unit 11, an operation receiving unit 12, a photographing control unit 13, a video photographing unit 14, a still image photographing unit 15, an abnormality detection unit 16, and a storage / readout unit 19. Each of these units is a function or means realized when any of the components shown in Fig. 6 operates in response to an instruction from the CPU 111 in accordance with a photographing device program loaded from the SRAM 113 onto the DRAM 114. The photographing device 10 also has a storage unit 1000 constructed by the ROM 112, SRAM 113, and DRAM 114 shown in Fig. 6. The storage unit 1000 stores the GUID (Globally Unique Identifier) ​​of the photographing device itself.

[0049] The transmitting / receiving unit 11 is an example of a transmitting means and a receiving means, and is mainly realized by the processing of the CPU 111, and communicates various data or information with other devices or terminals. Furthermore, the transmitting / receiving unit 11 uses the network I / F 121 to communicate data with other devices or terminals via the communication network 100.

[0050] The operation reception unit 12 is mainly realized by the processing of the CPU 111 on the operation unit 115, and receives various selections or inputs from the user who is the photographer.

[0051] The shooting control unit 13 is mainly realized by the processing of the CPU 111 on the imaging unit 101, image processing unit 104, and imaging control unit 105, and captures an image of a subject such as a landscape and acquires captured image data. The shooting control unit 13 performs shooting by, for example, switching between video shooting by the video shooting unit 14 and still image shooting by the still image shooting unit 15 in a time-division manner.

[0052] The moving image shooting section 14 is realized mainly by the processing of the CPU 111 on the imaging unit 101, the image processing unit 104, and the imaging control unit 105, and performs moving image shooting by the imaging device 10.

[0053] The still image capturing section 15 is mainly realized by the processing of the imaging unit 101, and captures an image of a subject such as a landscape, and captures a still image using the imaging device 10. The still image capturing section 15, for example, captures a still image (photograph) with a higher resolution than the video captured by the video capturing section 14, and stores the captured image data in the storage section 1000. The still image captured by the still image capturing section 15 may be a single frame image, or may be an HDR (High Dynamic Range) image composed of multiple images.

[0054] The abnormality detection unit 16 is mainly realized by the processing of the CPU 111, and performs various abnormality detection processes, which will be described later, from still images taken by the still image taking unit 15 and videos taken by the video taking unit .

[0055] The storage / readout unit 19 is an example of a setting means, and is mainly realized by the processing of the CPU 111, and stores various data (or information) in the storage unit 1000 and reads out various data (or information) from the storage unit 1000. The storage unit 1000 also stores image data captured by the video capture unit 14 and the still image capture unit 15. The image data stored in the storage unit 1000 is associated with the capture time of the captured image as metadata.

[0056] The storage unit 1000 has constructed therein an abnormality management DB 1001, an image management DB 1002, and a Twin management DB 1003. The abnormality management DB 1001 is configured by an abnormality management table, which will be described later, and stores and manages information on the abnormality detection process executed by the abnormality detection unit 16 for the captured data (still images, video images), etc.

[0057] The image management DB 1002 stores and manages the captured images captured by the imaging device 10 and the processed images obtained by performing image processing on the captured images by the anomaly detection unit 16, in association with the image IDs. The captured images and processed images may be stored in a storage medium such as an SD memory card, which is the external medium described above.

[0058] The Twin management DB 1003 manages Twin IDs that identify Twins in association with user IDs. Twin, which will be described in detail later, indicates a function of storing information such as metadata, configuration, and status related to actual devices such as the image capture device 10 in the server 50.

[0059] <Server functional configuration> Next, the functional configuration of server 50 will be described with reference to Fig. 8. Server 50 has a transmitting / receiving unit 51, a receiving unit 52, a determining unit 55, an abnormality detecting unit 56, and a storing / reading unit 59. Each of these units is a function or means realized when any of the components shown in Fig. 7 operates in response to an instruction from CPU 501 in accordance with a server program loaded from HD 504 onto RAM 503. Server 50 also has a storage unit 5000 constructed by ROM 502, RAM 503, and HD 504 shown in Fig. 7.

[0060] The transmitter / receiver 51 is an example of a transmitting means and a receiving means, and is mainly realized by processing of the CPU 501 on the network I / F 509, and transmits and receives various data or information to and from other devices or terminals via the communication network 100. The transmitter / receiver 51 receives (acquires), for example, a moving image captured by the camera device 10 from the camera device 10 or the communication terminal 90. The transmitter / receiver 51 also receives (acquires), for example, a still image captured by the camera device 10 from the camera device 10 or the communication terminal 90.

[0061] The reception unit 52 is mainly realized by the processing of the CPU 501 on the keyboard 511 or the pointing device 512, and receives various selections or inputs from the user.

[0062] The determination unit 55 is realized by the processing of the CPU 501, and performs various determination processes.

[0063] The abnormality detection unit 56 is mainly realized by the processing of the CPU 501, and performs various abnormality detection processes (described later) on the photographed data (still images and moving images).

[0064] The storage / reading unit 59 is an example of a setting means, and is mainly realized by processing of the CPU 501, and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000.

[0065] The storage unit 5000 has configured therein an abnormality management DB 5001, an image management DB 5002, and a twin management DB 5003. The abnormality management DB 5001 is configured by an abnormality management table, which will be described later, and stores and manages information on the abnormality detection process executed by the abnormality detection unit 56 for still images and videos.

[0066] The image management DB 5002 stores and manages the photographic data (still images, moving images) received from the photographing device 10 or the communication terminal 90 in association with the image ID.

[0067] The images in the image management DB 5002 may be transmitted to the photographing device 10 or the communication terminal 90 and stored in the image management DB 1002 or 9002 on the photographing device 10 or the communication terminal 90 side, or in a storage medium such as an SD memory card.

[0068] The Twin management DB 5003 manages a Twin ID for identifying a Twin in association with a user ID and a model ID for identifying the image capturing apparatus 10.

[0069] <Functional configuration of communication terminal> Next, the functional configuration of the communication terminal 90 will be described with reference to Fig. 8. The communication terminal 90 has a transmitting / receiving unit 91, a receiving unit 92, a display control unit 93, an abnormality detection unit 96, and a storage / reading unit 99. Each of these units is a function or means realized when any of the components shown in Fig. 8 operates in response to an instruction from the CPU 901 in accordance with a communication terminal program loaded from the HD 904 onto the RAM 903. The communication terminal 90 also has a storage unit 9000 constructed by the ROM 902, RAM 903, and HD 904 shown in Fig. 8.

[0070] The transmitter / receiver unit 91 is an example of a transmitting means and a receiving means, and is mainly realized by processing of the CPU 901 on the network I / F 909, and transmits and receives various data or information with other devices or terminals via the communication network 100.

[0071] The reception unit 92 is an example of a setting unit, and is mainly realized by the processing of the CPU 901 on the keyboard 911 or the pointing device 912, and receives various selections or inputs from the user.

[0072] The display control unit 93 is mainly realized by the processing of the CPU 901, and causes various images, characters, etc. to be displayed on the display 906. The display control unit 93 accesses the server 50 using, for example, a web browser or a dedicated application, and causes an image corresponding to data distributed from the server 50 to be displayed on the display 906.

[0073] The abnormality detection unit 96 is mainly realized by the processing of the CPU 901, and performs various abnormality detection processes, which will be described later, on still images and moving images.

[0074] The storage / readout unit 99 is mainly realized by the processing of the CPU 901 , and stores various data (or information) in the storage unit 9000 and reads out various data (or information) from the storage unit 9000 .

[0075] An abnormality management DB 9001 and an image management DB 9002 are constructed in the storage unit 9000. The abnormality management DB 9001 is configured by an abnormality management table, which will be described later, and stores and manages information on the abnormality detection process executed by the abnormality detection unit 96 for the captured data (still images, video images), etc.

[0076] The image management DB 9002 stores and manages the captured images captured by the image capturing device 10 and received from the image capturing device 10 in association with the image ID. The captured images and processed images may be stored in a storage medium such as an external storage medium, such as an SD memory card.

[0077] The image processing system 1 described above has a Twin function. The Twin function is a function for storing information such as metadata, configuration, and status related to a physical device such as the image capture device 10 in the server 50. As a result, a digital twin (twin) of the image capture device 10 is generated in the server 50.

[0078] The actual device (image capture device 10) and Twin (server 50) are synchronized via a communication network 100. The synchronized information also includes images (still images and videos) and information related to image capture.

[0079] The Twin function enables direct communication between the server 50 and the photographing device 10, so that image data captured by the photographing device 10 is directly uploaded and reflected on the Twin side, setting changes on the photographing device 10 side are reflected on the Twin side, and setting changes on the Twin side are reflected on the photographing device 10 side.

[0080] In Figure 8, the functional units that make server 50 function as a Twin are transmission / reception unit 51, reception unit 52, judgment unit 55, storage / readout unit 59, and memory unit 5000, and in Figure 7, the hardware that makes server 50 function as a Twin is CPU 501, ROM 502, RAM 503, network I / F 509, HDD controller 505, and HD 504.

[0081] FIG. 9 is a conceptual diagram showing an example of a Twin management table according to this embodiment.

[0082] FIG. 9 shows a Twin management table constituting the Twin management DB 5003 constructed in the storage unit 5000 of the server 50 shown in FIG.

[0083] In the Twin management table constituting the Twin management DB 5003, a Twin ID for identifying a Twin is managed in association with a user ID and a model ID for identifying the image capturing apparatus 10.

[0084] Note that the Twin management table constituting the Twin management DB 1003 constructed in the storage unit 1000 of the image capturing apparatus 10 shown in FIG. 8 also manages Twin IDs that identify Twins in association with user IDs.

[0085] Here, the user ID may be shared by a team of multiple people, and the user ID may be used as the team ID, with an ID that identifies an individual being linked to the user ID.

[0086] In FIG. 9, one or more user IDs and one photographing device are assigned to one Twin ID, but multiple Twin IDs may be assigned to one user ID.

[0087] FIG. 10 is a sequence diagram showing an example of the Twin registration process according to this embodiment.

[0088] The reception unit 92 of the communication terminal 90 receives the set information when the user inputs the user ID and Twin ID (step S501).

[0089] The transmitting / receiving unit 91 transmits the user ID and Twin ID accepted in step S501 to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the user ID and Twin ID transmitted from the communication terminal 90 (step S502).

[0090] The transmitter / receiver 91 of the communication terminal 90 transmits the user ID and Twin ID accepted in step S501 to the photographing device 10, and the transmitter / receiver 11 of the photographing device 10 receives the user ID and Twin ID transmitted from the communication terminal 90 (step S503).

[0091] The transmitter / receiver 11 of the photographing device 10 transmits the Twin ID and model ID received in step S503 to the server 50, and the transmitter / receiver 51 of the server 50 receives the Twin ID and model ID transmitted from the photographing device 10 (step S504).

[0092] The judgment unit 55 of the server 50 judges whether Twin registration is possible based on the user ID received in step S502 and the model ID received in step S504 (step S505). If registration is possible, the transmission / reception unit 51 transmits the user ID, Twin ID, and registration possibility information indicating that Twin registration is possible to the photographing device 10, and the transmission / reception unit 11 of the photographing device 10 receives the user ID, Twin ID, and registration possibility information transmitted from the server 50 (step S506).

[0093] Based on the registration possible information received in step S506, the storage / readout unit 19 associates the Twin ID and user ID received in step S506 and stores and registers them in the Twin management DB 1003 (step S507).

[0094] The transmitter / receiver 11 of the photographing device 10 transmits the user ID, Twin ID, model ID, and registration completion information indicating that the Twin ID has been registered to the server 50, and the transmitter / receiver 51 of the server 50 receives the user ID, Twin ID, model ID, and registration completion information transmitted from the photographing device 10 (step S508).

[0095] Based on the registration completion information received in step S508, the storage / readout unit 59 associates the Twin ID, model ID, and user ID received in step S508 and stores and registers them in the Twin management DB 5003 (step S509).

[0096] <<Spherical image generation>> Next, a method for generating a spherical image from images captured by a plurality of fisheye lenses will be described with reference to Figs. 11 to 17. This embodiment shows an example in which two fisheye lenses are used, but the present invention is not limited to this.

[0097] FIG. 11 is a diagram illustrating the projection relationship of the image capturing device 10 in the image processing system 1 according to this embodiment. An image captured with a single fisheye lens captures an image of approximately a hemisphere's worth of rotation from the capturing point. Furthermore, as shown in FIG. 11(a), the fisheye lens generates an image with an image height h corresponding to the angle of incidence φ relative to the optical axis. The relationship between the image height h and the angle of incidence φ is determined by a projection function according to a predetermined projection model. Furthermore, the embodiment described employs a so-called circular fisheye lens configuration, in which the image circle diameter is smaller than the image diagonal. The resulting partial image is a planar image that includes the entire image circle onto which approximately a hemisphere's worth of the capturing range is projected, as shown in FIG. 11(b).

[0098] Fig. 12 is a diagram illustrating the data structure of an equirectangular projection image, which is a celestial sphere image format used in this embodiment. As shown in Fig. 12, the equirectangular projection image is expressed as an array of pixel values ​​whose coordinates are a vertical angle φ corresponding to the angle with respect to a predetermined axis and a horizontal angle θ corresponding to the rotation angle around the axis. Each coordinate value (θ, φ) is associated with a point on a spherical surface that represents all directions centered on the shooting point, and the all directions are mapped onto the celestial sphere image.

[0099] <Connection position detection> FIG. 13 is a flowchart showing how to create a spherical image by stitching together images captured with two fisheye lenses.

[0100] The abnormality detection unit 16 performs distortion correction on the partial images A1 and B1 acquired by the image pickup elements 103a and 103b, respectively, using a position detection conversion table (step S121).

[0101] FIG. 14 is a diagram illustrating a conversion table for converting a fisheye image into a spherical image according to this embodiment. The conversion table defines the projection of a partial image expressed in a planar coordinate system onto an image expressed in a spherical coordinate system. As shown in FIGS. 14(a) and 14(b), the conversion table stores, for each fisheye lens, information that associates the coordinate values ​​(θ, φ) of a corrected image with the coordinate values ​​(x, y) of a partial image before correction that are mapped to the coordinate values ​​(θ, φ). In the example shown in FIG. 14, the angle covered by one pixel is 1 / 10 degrees in both the φ and θ directions, and the conversion table stores information indicating a 3600 × 1800 correspondence relationship for each fisheye lens. The position detection conversion table used when detecting a splice position is calculated and tabulated in advance by a manufacturer or the like after correcting distortion from an ideal lens model.

[0102] FIG. 15 is a diagram illustrating mapping of partial images captured by a fisheye lens onto a spherical coordinate system during position detection processing according to this embodiment. As a result of processing using the position detection conversion table, two partial images A1 and B1 captured by the fisheye lens are developed in a spherical image format as shown in FIG. 15. Partial image A1 captured through fisheye lens 102a is typically mapped onto approximately the upper hemisphere of the celestial sphere, and partial image B1 captured through fisheye lens 102b is mapped onto approximately the lower hemisphere of the celestial sphere. Corrected image a1 for position detection (also referred to as "corrected image a1") and corrected image b1 for position detection (also referred to as "corrected image b1"), which are expressed in the spherical format, extend beyond the respective hemispheres because the full angle of view of the fisheye lens exceeds 180 degrees. As a result, when corrected image a1 and corrected image b1 are superimposed, an overlapping region occurs where the captured ranges of the images overlap.

[0103] As a result, a corrected image a1 obtained by correcting the distortion of the partial image A1 and a corrected image b1 obtained by correcting the distortion of the partial image B1 are obtained.

[0104] The abnormality detection unit 16 or the abnormality detection unit 56 detects a joint position between the images in the overlapping region of the corrected image a1 and the corrected image b1 (step S122).

[0105] The seam position between the corrected images a1 and b1 is detected by pattern matching from the corrected images A1 and B1 converted using the seam position detection conversion table, and a seam position detection result is generated. In the position detection conversion table according to this embodiment, as shown in FIG. 14 , the optical axes of the two lens optical systems are projected onto two poles of a sphere, and the overlapping area between the images is projected near the equator of the sphere. In a spherical coordinate system, the closer the vertical angle φ is to the pole where it is 0 degrees or 180 degrees, the greater the distortion, degrading the accuracy of seam position detection. In contrast, by using the projection described above, the accuracy of seam position detection can be improved. While FIG. 14 illustrates the mapping of two partial images captured using two fisheye lenses onto a spherical coordinate system, three or more fisheye lenses may also be used.

[0106] 16 is a diagram illustrating the connection position detection process. Template image 520 is an image of the overlapping area of ​​corrected image b1, and search image 530 is an image of the overlapping area of ​​corrected image a1. Here, assuming that a template image is generated with a specified size W and a specified generation interval, multiple template images 520-1 to 520-# are generated in the manner shown in FIG.

[0107] For the generated plurality of template images 520-1 to 520-#, a corresponding portion 540-1 on search image 530 is searched for within a predetermined search range by template matching. For each of template images 520-1 to 520-#, the amount of shift from the reference position to the position at which the matching score is maximized is detected.

[0108] The anomaly detection unit 16 or the anomaly detection unit 56 receives the corrected images a1 and b1 converted in step S121, detects the connection positions between the input corrected images a1 and b1 by pattern matching processing, and generates detection result data.

[0109] The abnormality detection unit 16 or the abnormality detection unit 56 uses the detection result data generated in step S122 to correct the position detection conversion table so that the image is aligned on the spherical coordinate system (step S123).

[0110] Since the shift amount is obtained for each coordinate value of the spherical image format by the joint position detection process in step S122, in step S123, the detection distortion correction table Ta used for the distortion correction of partial image A1 is corrected so that the input coordinate value (θ, φ) is correlated with (x, y), which was previously correlated with (θ+Δθ, φ+Δφ). Note that there is no need to change the correlation in the detection distortion correction table Tb used for the distortion correction of partial image B1.

[0111] The abnormality detection unit 16 or the abnormality detection unit 56 generates a conversion table for image synthesis by performing rotational coordinate conversion on the conversion table for position detection corrected in step S123 (step S124).

[0112] The abnormality detection unit 16 or the abnormality detection unit 56 uses the conversion table for image synthesis generated in step S124 to perform distortion correction on the original partial image A1 and partial image B1, thereby obtaining a corrected image for image synthesis A2 and a corrected image for image synthesis B2 (step S125).

[0113] FIG. 17 is a diagram illustrating mapping of partial images captured by fisheye lenses onto a spherical coordinate system during image synthesis processing. The rotational coordinate transformation converts the definition of the coordinate axes of the horizontal and vertical angles with respect to the optical axis of one of the lens optical systems, as shown in FIG. 15, into definitions of the horizontal and vertical angles with respect to an axis perpendicular to the optical axis, as shown in FIG. 17. As a result, two partial images A1 and B1 captured by the fisheye lenses are developed onto a celestial sphere image format, as shown in FIG. 17. Corrected image A2 for image synthesis, which has been captured through fisheye lens 102a and has undergone distortion correction, is typically mapped onto approximately the left hemisphere of the celestial sphere, and corrected image B2 for image synthesis, which has been captured through fisheye lens 102b and has undergone distortion correction, is mapped onto approximately the right hemisphere of the celestial sphere. Note that while FIG. 17 illustrates mapping of two partial images captured by two fisheye lenses onto a spherical coordinate system, in the case of three or more fisheye lenses, three or more partial images are synthesized to generate a celestial sphere image.

[0114] The abnormality detection unit 16 or the abnormality detection unit 56 combines the corrected image for image synthesis A2 and the corrected image for image synthesis B2 (step S126).

[0115] In this synthesis process, blending or the like is performed on overlapping areas where the images overlap, and the existing pixel values ​​are used for areas where only one of the images has pixel values. Through the above synthesis process, one omnidirectional image is generated from two partial images captured by a fisheye lens.

[0116] <<Anomaly detection>> Various abnormalities can occur in the imaging device 10 that can interfere with image capture. If an abnormality causes the imaging device 10 to stop working and prevent image capture, the user may be able to notice the abnormality in the imaging device 10. However, if image capture is possible despite the abnormality, the user may continue to capture images without noticing the abnormality. Furthermore, various abnormalities can occur in the imaging device 10, and it is extremely difficult for the user to notice each abnormality. If the user continues to capture images without noticing the abnormality, problems may occur with the captured still images and videos, making the captured data unusable. This may necessitate recapturing under the same conditions. If a large amount of data has already been captured, or if recapturing under the same conditions is not possible, the damage can be immeasurable.

[0117] Anomaly detection by the imaging device 10 according to this embodiment will be described below.

[0118] <Abnormal item> There are a wide variety of abnormalities that can occur in the imaging device 10. For example, lens-related abnormalities include one-sided blur, lens scratches, coating peeling, peripheral light intensity, dust on the imaging element, and foreign matter attached to the lens.

[0119] One-sided blur refers to a state in which resolution is reduced when comparing one or more lenses. Generally, resolution tends to be lower at the periphery of a wide-angle lens (such as a fisheye lens) compared to the center, but the degree of reduction is about the same when comparing the periphery of the lens. If the lens condition changes during transportation after leaving the factory or due to impact during use, the resolution may be reduced more than expected in some parts of the periphery of the lens. Also, if multiple lenses are used, it is possible that the resolution of a specific lens will be reduced when comparing the individual lenses.

[0120] Lens scratches refer to scratches on the lens caused by dropping, tipping over, or hitting the image capture device 10 against an object. In particular, if the lens has a protruding structure or multiple lenses, it is possible that scratches may be unintentionally caused on the lens. If a lens scratch occurs, it may cause partial image defects, blurring, bleeding, flare, ghosting, etc., in the same position in the captured image every time a photograph is taken with the image capture device 10.

[0121] Coat peeling is a phenomenon in which a thin film formed on the lens surface to improve and protect lens performance peels off, and occurs due to contact with objects or deterioration over time. Coat peeling can cause blurring, bleeding, flare, ghosting, and other issues in captured images. Peripheral illumination refers to the amount of light at the periphery of the lens relative to the center. A decrease in the amount of light entering the periphery compared to the center of the lens causes the image in the area corresponding to the lens periphery to become darker. This generally occurs with wide-angle lenses (fisheye lenses, etc.), but this phenomenon is reduced by image processing and other methods. If the lens condition has changed since shipping from the factory, the expected correction may not be made and the phenomenon may worsen.

[0122] Image sensor dust refers to a state in which dust has adhered to the surface of the image sensor (103a, 103b). Although the exterior of the photographing device 10 is sealed by screws or the like, dust can get in through gaps or dust generated inside the photographing device 10 can adhere to the surface of the image sensor. If dust adheres to the surface of the image sensor, a foreign object will unintentionally appear in the same position in the photographed image every time a photograph is taken with the photographing device 10. Depending on the photographing settings of the photographing device 10, dust may be noticeable (large aperture value) or not (small aperture value).

[0123] Foreign matter adhesion to a lens refers to a state in which foreign matter such as dirt, dust, fingerprints, or water droplets adheres to the lens surface. In particular, when the lens has a protruding structure or is equipped with multiple lenses, there are many opportunities for the lens to come into contact with foreign matter, and it is possible that foreign matter may adhere to the lens without the lens noticing. If foreign matter adheres to the lens, the foreign matter will unintentionally appear in the same position in the captured image every time a photograph is taken with the imaging device 10.

[0124] In this embodiment, an example of a lens-related abnormality will be described, but the present invention is not limited to this and can be applied to any abnormality that occurs in an imaging device.

[0125] First abnormality detection process <Anomaly detection sequence> 18 and 19 are sequence diagrams showing the first anomaly detection process according to this embodiment. Here, an example is shown in which anomaly detection is set in the communication terminal 90, and anomaly detection is performed by the image capture device 10 and the server 50.

[0126] The user operates the reception unit 92 of the communication terminal 90 to enter an anomaly detection setting mode and inputs identification information (user ID, model ID, TwinID, etc.) to receive the set information (step S1). The transmission / reception unit 91 of the communication terminal 90 transmits the identification information and a request for an anomaly detection setting file for the camera device 10 to the camera device 10, and the transmission / reception unit 11 of the camera device 10 receives the identification information and the request for an anomaly detection setting file for the camera device 10 transmitted from the communication terminal 90 (step S2). The storage / readout unit 19 searches for and reads out the anomaly detection setting file corresponding to the identification information stored in the anomaly management DB 1001 based on the identification information and the request for an anomaly detection setting file for the camera device 10 received in step S2 (step S3). The anomaly detection setting file is a file containing anomaly detection setting information, and details of this file will be described below.

[0127] <Anomaly detection configuration file> Fig. 20 is a diagram showing an example of an anomaly detection setting file. Fig. 20(a) shows the setting items and setting contents of the anomaly detection setting file. Each device (including terminals and servers) stores an anomaly detection setting file in its own device. The anomaly detection setting file shown in Fig. 20 exists for each different imaging device. The setting items for each imaging device include detection items, setting authority, detection device, and error notification.

[0128] The detection items are items for which abnormalities are detected. For example, lens-related detection items include one-sided blur, lens scratches, coating peeling, peripheral light intensity, image sensor dust, and foreign matter attached to the lens. Predefined (prepared) abnormality detection items (see Figure 24) are set.

[0129] The setting authority indicates the authority to set and change the anomaly detection settings. For example, in the case of "communication terminal + server," the settings of each item (detection device, error detection) related to the detection item "one-sided blur" in the anomaly detection file can be changed from the communication terminal 90 and the server 50, but the settings of each item related to the detection item "one-sided blur" in the anomaly detection file cannot be changed from the imaging device 10 and the external server 70.

[0130] The detection device indicates a device that performs abnormality detection.

[0131] The error notification indicates the destination of the error notification when an abnormality is detected during abnormality detection.

[0132] In the case of Fig. 20(a), the settings for setting authority, detection device, and error notification are made by selecting from the image capture device 10, communication terminal 90, server 50, and external server 70, and multiple selections are also possible. For example, as shown in Fig. 20(b), each item may be defined by a number and the combination may be set by the sum of the numbers (1 (image capture device) + 4 (server) = 5 (image capture device + server)). Fig. 20(a) shows that when the detection item is one-sided blur, there is authority to set and change the setting when accessed by the image capture device and the server, and an abnormality is detected by the image capture device and an error notification is sent to the image capture device.

[0133] "Photographing device + server" refers to a case where the photographing device 10 and the server 50 perform cooperative processing. Here, the photographing device 10 performs anomaly detection ("simple" detection or first anomaly detection), and the server 50 performs further anomaly detection ("detailed" detection or second anomaly detection), thereby cooperatively performing anomaly detection. For example, the local photographing device 10 detects anomalies by comparing the hemispherical image data shown in FIGS. 2(a) and 2(b) before creating the photographed image shown in FIG. 2(c) and transmitting it to the server 50 as photographed data. The server 50 then performs further anomaly detection using the photographed data. Note that in order to reduce the processing load on the photographing device 10 and thereby suppress the amount of heat generated by the photographing device 10, the server 50 may perform the first anomaly detection and the second anomaly detection by synthesizing the hemispherical image data shown in FIGS. 2(a) and 2(b) to create a photographed image.

[0134] The error notification setting file must be saved in association with identification information (user ID, model ID, TwinID, etc.). Therefore, the identification information may be included in the file name of the error notification setting file, or may be recorded within the file. The data may be stored in a known data format such as JSON (JavaScript Object Notation) or CSV (Comma Separated Values), or may be stored in a binary format.

[0135] If there is no corresponding anomaly detection setting file in the anomaly management DB 1001, an anomaly detection setting file corresponding to the identification information used in the search is created and stored in the anomaly management DB 1001. In this case, the settings in the anomaly detection setting file are set to predefined default settings.

[0136] The transmitter / receiver unit 11 of the photographing device 10 transmits the identification information received in step S2 and the abnormality detection setting file of the photographing device 10 to the communication terminal 90, and the transmitter / receiver unit 91 of the communication terminal 90 receives the identification information and the abnormality detection setting file of the photographing device 10 transmitted from the photographing device 10 (step S4).

[0137] Next, the transmitting / receiving unit 91 of the communication terminal 90 transmits the identification information and a request for an anomaly detection setting file of the server 50 to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the identification information and the request for an anomaly detection setting file of the server 50 transmitted from the communication terminal 90 (step S5). The storage / reading unit 59 searches for and reads out the anomaly detection setting file corresponding to the identification information stored in the anomaly management DB 5001 from the identification information and the request for an anomaly detection setting file of the server 50 received in step S5 (step S6).

[0138] If there is no corresponding anomaly detection setting file in the anomaly management DB 5001, an anomaly detection setting file corresponding to the identification information used in the search is created and stored in the anomaly management DB 5001. In this case, the settings in the anomaly detection setting file are set to predetermined default settings.

[0139] The transceiver 51 of the server 50 transmits the identification information received in step S2 and the abnormality detection setting file of the server 50 to the communication terminal 90, and the transceiver 91 of the communication terminal 90 receives the abnormality detection setting file of the server 50 transmitted from the server 50 (step S7).

[0140] <Anomaly detection setting screen> 21 is a diagram showing an example of an abnormality detection setting screen, which is displayed on the LCD 122 in the case of the image capturing device 10 and on the display 906 in the case of the communication terminal 90.

[0141] The items on the anomaly detection setting screen match the contents of the anomaly detection setting file. The anomaly detection setting file received from the imaging device 10 in step S4 is compared with the anomaly detection setting file received from the server 50 in step S7. If there are any differences, a setting priority order (e.g., server > imaging device) is determined in advance, and the anomaly detection setting file with the higher priority is used. The selection information on the anomaly detection setting screen is displayed based on the anomaly detection setting file. In Figure 21, the setting authority item is displayed differently depending on whether the device displaying the anomaly detection setting screen has setting and change authority; if the device has change authority, it displays "Changeable," and if the device does not have change authority, it displays "Cannot be changed." If the setting authority is "Cannot be changed," the options cannot be changed.

[0142] <Detection device> Because there are limitations on detection devices and optimal devices (detection locations) and methods depending on the detection item, it is advisable to define detectable devices for each detection item in advance and allow the user to select from the detectable devices. In this case, the displayed detection devices will differ depending on the detection item. Detection devices include the camera 10, the communication terminal 90, and the server 50. Detection by the camera 10 is suitable for processes that can only be performed by a processing step of the camera 10, simple processes, and processes that require urgency. Detection by the communication terminal 90 is suitable for processes that cannot be performed by the camera 10 or that utilize functions of the communication terminal 90, acting as an intermediary between the camera 10 and the server 50. Detection by the server 50 is suitable for processes that take time for the camera 10 or the communication terminal 90 (such as high-precision processes for similar processes) and complex processes that cannot be performed with the resources of the camera 10 or the communication terminal 90.

[0143] Also, a case can be considered in which simple anomaly detection is performed by the image capture device 10 and detailed anomaly detection is performed by the server 50. Another case can be considered in which the image capture device 10 detects anomalies based on the results of anomaly detection by the server 50, thereby improving detection accuracy. Therefore, multiple detection devices may be selectable, or multiple detection levels may be prepared instead of detection devices as shown in FIG. 22 , and the detection level and the number of detection times for "simple" and "detailed" may be fixed and set in advance, such that, for example, "simple" detects anomalies in one location and "detailed" detects anomalies in multiple devices. In this case, for example, when "simple" is selected, anomaly detection by a single device (e.g., the server 50) is automatically set, and when "detailed" is selected, anomaly detection by cooperation between multiple devices (e.g., the image capture device 10 and the server 50) is automatically set.

[0144] <Error notification> For error notification, an error notification destination is set when an abnormality is detected. In FIG. 21 , the options are the image capture device 10, the communication terminal 90, the server 50, and the external server 70, and when an abnormality is detected, error information is displayed at the error notification destination. For example, in the case of the image capture device 10, an error is displayed on the LCD 122, in the case of the communication terminal 90, on the display 906, and in the case of the server 50, on the display 506. Also, in FIG. 21 , an error is displayed by setting a device as the error notification destination, but this is not limiting. Alternatively, several error notification means may be selected, such as registering an e-mail address in advance and sending an e-mail to a specified address when an abnormality is detected, or sending an electronic voice message to a specific telephone number.

[0145] The user selects settings for each item on the anomaly detection setting screen using the reception unit 92 to perform anomaly detection settings, receives the various set information, and creates an anomaly detection setting file common to the imaging device 10 and the server 50 (step S8).

[0146] The transmitting / receiving unit 91 transmits the identification information and the anomaly detection setting file created in step S8 to the imaging device 10, and the transmitting / receiving unit 11 of the imaging device 10 receives the identification information and the anomaly detection setting file transmitted from the communication terminal 90 (step S9). The storage / reading unit 19 associates the identification information and the anomaly detection setting received in step S9 with each other and stores them in the anomaly management DB 1001 (step S10).

[0147] Similarly, the transmitting / receiving unit 91 transmits the identification information and the anomaly detection setting file created in step S8 to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the identification information and the anomaly detection setting file transmitted from the communication terminal 90 (step S11). The storage / reading unit 59 associates the identification information and the anomaly detection setting file received in step S11 with each other and stores them in the setting information management DB 5001 (step S12).

[0148] <Detection of abnormalities in imaging and imaging equipment> When the user gives a shooting instruction from the shooting control unit 13, the shooting control unit 13 reads the abnormality detection setting from the abnormality detection management DB 1001 using the identification information as a search key using the memory / readout unit 19, and shoots a still image or a video using the still image shooting unit 15 or the video shooting unit 14 (step S13).The memory / readout unit 19 then associates the shooting data indicating the shot still image or video with the image ID and stores it in the image management DB 1002 (step S14).

[0149] The abnormality detection unit 16 executes abnormality detection and notification (display) on the imaging device side for the imaging data captured in step S13 in accordance with the abnormality detection setting (step S15), and the storage / readout unit 19 stores information on the abnormality detected by the abnormality detection unit 16 as an abnormality management file in the abnormality management DB 1001 (step S16). The abnormality detection process in step S15 may be performed on the imaging data stored in the image management DB 1002, or may be performed on intermediate data in the imaging sequence of step S13.

[0150] <Abnormality management file> 23 is a diagram showing an example of an abnormality management file according to this embodiment. The abnormality management file records information such as date and time, error code, image ID, detection device, and detection data. Each item will be explained below.

[0151] The date and time when the abnormality was detected is recorded.

[0152] The error code records the result of an anomaly detection. Error codes are defined in advance for each anomaly detection item, and the error code corresponding to the result of an anomaly detection is recorded.

[0153] FIG. 24 is a diagram showing an example of an error code for each abnormality detection item.

[0154] Figure 24(a) shows an example of an error code defined for each abnormality detection item. The abnormality detection items include one-sided blur, lens scratches, coating peeling, peripheral light intensity, image sensor dust, and foreign matter attached to the lens. Anomaly detection is performed for each item, and error codes are defined in advance for each abnormality detection item, such as E1000 if one-sided blur is detected, and E2000 if a lens scratch is detected. An error code (E0000) is also defined for when no abnormality is detected.

[0155] Although the error codes shown in FIG. 24(a) may be used, the error codes shown in FIG. 24(b) may also be used. FIG. 24(b) is an example in which error codes are defined in more detail than those in FIG. 24(a). The error codes are divided into upper and lower levels, with the upper level indicating the abnormality detection item and the lower level defining more detailed content within the abnormality detection item. For example, in the case of error code E1010, the upper level E10 indicates the detection of one-sided blur, and the lower level 10 indicates the lens 102a side, thereby indicating that one-sided blur has been detected on the lens 102a side.

[0156] The image ID records the image information used when anomaly detection was performed. If no image information is used, you can leave this field blank.

[0157] The detection device column records the detection device used when an anomaly was detected. Anomaly detection is performed using the device configured in the anomaly detection configuration file, and the device that detected the anomaly is recorded. The detection devices shown in Figure 20(b) can be defined by numbers, and the combination can be recorded as the sum of the numbers. Depending on the detection item, detection may be performed at multiple locations, so the results of detection at each location can also be recorded separately. For example, Figure 23 shows that for the image with image ID P0010, a lens scratch (error code E2000) was detected by both the camera and the server. The detected data is recorded as the detected data calculated when an abnormality is detected.

[0158] FIG. 25 is a diagram illustrating a recording format for detection data. FIG. 25(a) is a format for recording detection data output for one-sided blur detection, in which, for example, the amount of blur for each lens is recorded as detection data. FIG. 25(b) is a format for recording detection data output for lens scratch detection, in which, for example, the lens scratch determination level for each lens is recorded as detection data. FIG. 25(c) is a combination of the recording formats shown in FIGS. 25(a), 25(b), etc., and shows a format that also includes detection data for other abnormality detections, such as coating peeling, peripheral light intensity, image sensor dust, and foreign matter attached to the lens, whose description has been omitted.

[0159] Since the detection data calculated differs depending on the abnormality detection item, the recording format can be changed for each abnormality detection item, or the detection data can be recorded in the section for the item where an abnormality was detected using a common recording format that includes all abnormality detection items.

[0160] The abnormality detection setting file and the abnormality management file stored in the abnormality management DB 1001 of the imaging device 10 and the abnormality management DB 5001 of the server 50 may be held in separate files, or two files with the same identification information may be stored and held in a single file.

[0161] <Error flag> Additionally, the anomaly management file may be configured to allow a single error flag to be set within the file, separate from the above. This flag indicates whether an anomaly has been detected, separate from the individual anomaly detection results. By checking the error flag, it is possible to immediately determine whether an anomaly has occurred in the imaging device 10 without checking the individual anomaly detection results. When an anomaly is detected, the error flag is set to ON (e.g., 1) in the anomaly management file. If an error flag is already ON and another anomaly is detected, the error flag remains ON. When all anomalies are resolved (e.g., due to part or unit replacement, maintenance, etc.), the error flag is set to OFF (e.g., 0). When the error flag is ON, it is recommended to issue an error notification or change the control of the imaging device. If the method for setting the error flag to OFF is kept secret from general users and disclosed only to specific members (e.g., the manufacturer, support service, etc.), the specific members will receive an inquiry when an anomaly is detected, and the anomaly can be resolved with the correct response.

[0162] <Detection of abnormalities in the imaging device 10> Detection of one-sided blur As an example of anomaly detection in step S15 by the image capture device 10, one-sided blur detection will be described. As described above, one-sided blur refers to a state in which resolution is reduced when comparing images captured by one or more lenses. In this embodiment, a method for detecting reduced resolution by comparing images captured by multiple lenses is described. The image capture device 10 is equipped with two fisheye lenses (102a, 102b). Using an overlapping area where the image capture ranges of the two fisheye lenses overlap, as shown in FIG. 16, a stitching position is detected and two partial images captured by the two fisheye lenses are stitched together. Because the same subject is captured in the overlapping area, the side of the two fisheye lenses where blur is occurring can be detected by comparing the images at the matching position using template matching for stitching position detection. For example, in FIG. 16, multiple template images (502-1 to 502-#) are generated in the overlapping area of ​​the position detection corrected image b1, and the overlapping area of ​​the position detection corrected image a1 is used as a search image to detect the joining position by template matching, and an area of ​​the same size as the template image is extracted from the search image at the matching position.

[0163] Next, the brightness variance is calculated for each of the images extracted from the template image and the search image. While this also affects the captured subject, a high variance indicates a complex (characteristic) subject, while a low variance indicates a simple (characteristic) subject. Blurred images tend to have a low variance because the blur prevents the capture of detailed subject features. Therefore, by comparing the variances of two images capturing the same subject, if there is an extreme difference in variance, it is possible to detect blurring on the side with the smaller variance. When multiple template images are used, as in Figure 16, variances are calculated for multiple regions. Therefore, thresholds can be set for the difference in variances and the number of detected anomalies. One-sided blurring can be determined to have occurred when the number of detected anomalies exceeds the threshold value. The detection data (blur amount in Figure 25(a)) in the anomaly management file in Figure 23 can also include the calculated variances for each image. If a one-sided blur anomaly is detected, a corresponding error code is recorded.

[0164] <Anomaly detection file synchronization> 19 , the transmitter / receiver 11 of the image capturing device 10 transmits the identification information, the captured image data (still image, video, etc.), and the anomaly detection file to the server 50 when triggered by a predetermined condition, and the transmitter / receiver 51 of the server 50 receives the identification information, the captured image data, and the anomaly detection file transmitted from the image capturing device 10 (step S17). Here, the predetermined condition that serves as the trigger is when the image capturing device 10 becomes able to connect (synchronize) with the server 50 via the communication network 100. For example, this may be when the image capturing device 10 cooperates with the communication terminal 90 (via the communication terminal 90) and connects to the server 50 via the communication network 100, or when the image capturing device 10 reconnects to the server 50 after being disconnected from the communication network 100 by a user or being unable to connect to the communication network 100 due to the communication environment. Note that if no anomaly is detected in step S15, the anomaly detection file records that there is no anomaly, or records nothing at all.

[0165] The storage / readout unit 59 stores the photographic data (still image, video, etc.) received in step S17 in the image management DB 5002 in association with the identification information (step S18).

[0166] The storage / reading unit 59 searches for and reads out the abnormality detection file corresponding to the identification information stored in the abnormality management DB 5001 from the identification information received in step S17, compares the abnormality detection file of the imaging device 10 received in step S17 with the abnormality detection file of the server 50 stored in the previous step S24, and if there is a difference, generates an abnormality detection file by complementing the two files and stores it in the abnormality management DB 5001 (step S19).

[0167] If there is a difference between the abnormality detection files of the photographing device 10 and the server 50 in step S19 and a supplemented abnormality detection file is created, the transmission / reception unit 51 transmits the identification information and the abnormality detection file supplemented (created) in step S19 to the photographing device 10, and the transmission / reception unit 11 of the photographing device 10 receives the identification information and the abnormality detection file transmitted from the server 50 (step S20).

[0168] The abnormality detection unit 16 checks the abnormality detection file received in step S20, and if an abnormality error code detected by the server 50 is recorded, the memory / readout unit 19 reads out the abnormality detection setting set in step S8 and stored in the abnormality management DB 1001 in step S10, and if an error notification setting is set in the imaging device, notifies (displays) the abnormality detection information (error content, error code, etc.) on the LCD 122 of the imaging device 10, etc. (step S21).

[0169] The storage / readout unit 19 associates the identification information received in step S20 with the abnormality detection file and stores them in the abnormality management DB 1001 (step S22).

[0170] <Server-side anomaly detection> The abnormality detection unit 56 reads the abnormality detection setting from the abnormality management DB 5001 using the identification information as a search key via the memory / readout unit 59, and if the detection device is set to the server in the abnormality detection setting, executes server-side abnormality detection processing on the shooting data (still images, videos, etc.) received from the shooting device 10 in step 18 and stored in the image management DB 5002 in association with the identification information (step S23), and the memory / readout unit 59 stores the information detected as an abnormality by the abnormality detection unit 56 in the abnormality management DB 5001 as an abnormality management file (step S24).

[0171] <Detection on Server 50> (Lens scratch detection) As an example of anomaly detection by server 50 in step S23, lens scratch detection will be described. As described above, lens scratches refer to scratches on the lens caused by dropping, tipping over, or hitting an object with image capture device 10. If a lens scratch occurs, there is a possibility that partial imaging defects, blurring, bleeding, flare, ghosting, etc. will occur at the same position in the captured image every time an image is taken with image capture device 10.

[0172] This embodiment describes a method for detecting lens scratches using machine learning. Lens scratches can cause a variety of symptoms depending on the location, size, shape, etc. of the scratch, making it extremely difficult to identify them from image data. If the scratch (the impact of the scratch) is significant, the user may notice it by looking at the lens or the image data and contact the manufacturer or support center for an inquiry or repair request. In such a case, it is possible to determine whether a lens scratch has occurred based on the problematic image capture device 10 and the image data. Since manufacturers and support centers accumulate image data due to lens scratches, this data is used as training data for machine learning to create an inference model. Furthermore, the image management DB 5002 of the server 50 stores image data (still images, videos, etc.) from the image capture device 10 in association with identification information (user ID, model ID, Twin ID, etc.). Image data from multiple image capture devices 10 is accessible to anyone with viewing permissions. Image data from lens scratches that are difficult for users to notice themselves is also included. Learning using this image data also enables the creation of a more accurate inference model. Furthermore, because new photographic data is added daily to the image management DB 5002 of the server 50, periodic machine learning can update the inference model to the latest, highly accurate state. The memory / readout unit 59 searches for and reads photographic data corresponding to the identification information stored in the image management DB 5002, and the anomaly detection unit 56 uses the inference model to detect lens scratches from the photographic data. The detection data (lens scratches in FIG. 25(b)) in the anomaly management file of FIG. 23 may record an inferred value inferred by the inference model. If a lens scratch anomaly is detected, a corresponding error code is recorded. Machine learning can be used to learn and infer not only lens scratches but also coating peeling, image sensor dust, and foreign matter attached to the lens. By learning image data with each problem as training data and adding it to the inference model, it is possible to classify and infer which anomaly has occurred in the image data for which anomaly detection is desired (classification).

[0173] <Anomaly detection in cooperation between the imaging device 10 and the server 50> (Lens scratches) An example of anomaly detection performed in cooperation between the image capture device 10 in step S15 and the server 50 in step S23 will be described below using lens scratch detection. Regarding lens scratch detection in the server 50, as described above, it has been explained that anomalies can be detected by creating the latest, highly accurate inference model using the image capture data stored in the image management DB 5002 of the server 50 and performing inference. Similarly, the image capture device 10 in step S15 can also detect lens scratches using machine learning. An inference model is stored in the image capture device 10, and an image is captured by the image capture device 10 (step S13), and lens scratches are detected by inference from the image capture data stored in the image management DB 1001 (step S14).

[0174] However, because the image capture device 10 and the server 50 have different resources, it is difficult to perform similar processing. It is desirable for the image capture device 10 to have light and simple processing. Therefore, unlike those performed by the server 50, the machine learning method and inference model are not frequently updated, and the inference model may be updated only when the image capture device 10 firmware is updated. The purpose of performing anomaly detection in the image capture device 10, even if it is simple, is to promptly notify the user of a possible anomaly at the time of image capture. Even if the anomaly detection is incorrect, the user can continue capturing images after checking the image capture device 10. The image capture device may be recorded in the detection device field of the anomaly management file in FIG. 23, and the detection data (lens scratch in FIG. 25(b)) may contain an inferred value inferred by the inference model.

[0175] 20(a), if multiple locations are set in the detection device field, if an abnormality is detected in one location, the other locations will also be detected. For example, if an abnormality is detected in the imaging device 10, the server 50 will also detect the abnormality. If the detection devices are different, the same detection method may be used, but the detection results will be the same, so if a different detection method is available, it is better to change it, and if a method of detection can be used later, it is even better if it can detect with high accuracy.

[0176] <Control changes when an abnormality is detected> When an abnormality is detected, in addition to notifying an error, the control of the image capture device 10 may be changed. For example, when the image capture device 10 captures an image (step S13) and stores the image in the image management DB 1001 (step S14), the image capture device 10 may output and store the data captured with the fisheye lens as is, without stitching the captured image data. By doing so, the user can recognize that something unusual has occurred when viewing the captured image data and find that it is recorded as unstitched fisheye lens image data. This allows the user to know from which image the abnormality was detected, and allows detection processing that can only be achieved by a processing step of the image capture device 10 to be performed elsewhere. Furthermore, depending on the type of abnormality, it may be possible to correct or mitigate the captured image data through image processing.

[0177] ● Second anomaly detection process 26 and 27 are sequence diagrams showing the second abnormality detection process according to this embodiment. Here, an example is shown in which abnormality detection is set in the image capture device 10, and abnormality detection is performed by the image capture device 10 and the server 50.

[0178] The user operates the operation reception unit 12 of the image capturing device 10 to enter an anomaly detection setting mode, and inputs identification information (user ID, model ID, TwinID, etc.) to receive the set information (step S25-1). The transmission / reception unit 11 of the image capturing device 10 transmits the identification information and a request for an anomaly detection setting file from the server 50 to the server 50, and the transmission / reception unit 51 of the server 50 receives the identification information and the request for an anomaly detection setting file from the server 50 transmitted from the image capturing device 10 (step S25-2). The storage / readout unit 59 searches for and reads out an anomaly detection setting file corresponding to the identification information stored in the anomaly management DB 5001 based on the identification information and the request for an anomaly detection setting file from the server 50 received in step S25-2 (step S25-3).

[0179] If there is no corresponding anomaly detection setting file in the anomaly management DB 5001, an anomaly detection setting file corresponding to the identification information used in the search is created and stored in the anomaly management DB 5001. In this case, the settings in the anomaly detection setting file are set to predefined default settings.

[0180] The transceiver 51 of the server 50 transmits the identification information received in step S25-2 and the abnormality detection setting file of the server 50 to the photographing device 10, and the transceiver 11 of the photographing device 10 receives the identification information and the abnormality detection setting file of the server 50 transmitted from the server 50 (step S25-4).

[0181] The storage / reading unit 19 searches for and reads out the abnormality detection setting file corresponding to the identification information stored in the abnormality management DB 1001 (step S25-5).

[0182] If there is no corresponding anomaly detection setting file in the anomaly management DB 1001, an anomaly detection setting file corresponding to the identification information used in the search is created and stored in the anomaly management DB 1001. In this case, the settings in the anomaly detection setting file are set to predefined default settings.

[0183] The abnormality detection setting screen of FIG. 21 is displayed on the LCD 122, and the user performs abnormality detection setting by selecting settings for each item on the abnormality detection setting screen using the operation reception unit 12, and various pieces of set information are received, and an abnormality detection setting file common to the imaging device 10 and the server 50 is created (step S25-6).

[0184] The storage / readout unit 19 associates the identification information created in step S25-6 with the abnormality detection setting file and stores them in the abnormality management DB 1001 (step S25-7).

[0185] The transmitting / receiving unit 11 transmits the identification information and the anomaly detection setting file created in step S25-6 to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the identification information and the anomaly detection setting file transmitted from the imaging device 10 (step S25-8). The storage / reading unit 59 associates the identification information and the anomaly detection setting received in step S25-8 with each other and stores them in the anomaly management DB 5001 (step S25-9).

[0186] Step S13 and subsequent steps are the same as those in the first embodiment, and therefore will not be described here.

[0187] ●Third abnormality detection process 28 and 29 are sequence diagrams showing the third abnormality detection process according to this embodiment. Here, an example is shown in which abnormality detection is set in the server 50, and abnormality detection is performed by the image capturing device 10 and the server 50. The user operates the reception unit 52 of the server 50 to enter an anomaly detection setting mode and inputs identification information (user ID, model ID, TwinID, etc.) to receive the set information (step S26-1). The transmission / reception unit 51 of the server 50 transmits the identification information and a request for an anomaly detection setting file for the camera device 10 to the camera device 10, and the transmission / reception unit 11 of the camera device 10 receives the identification information and the request for an anomaly detection setting file for the camera device 10 transmitted from the server 50 (step S26-2). The storage / readout unit 19 searches for and reads out an anomaly detection setting file corresponding to the identification information stored in the anomaly management DB 1001 based on the identification information and the request for an anomaly detection setting file for the camera device 10 received in step S26-2 (step S26-3).

[0188] If there is no corresponding anomaly detection setting file in the anomaly management DB 1001, an anomaly detection setting file corresponding to the identification information used in the search is created and stored in the anomaly management DB 1001. In this case, the settings in the anomaly detection setting file are set to predefined default settings.

[0189] The transmitter / receiver 11 of the photographing device 10 transmits the identification information and the abnormality detection setting file of the photographing device 10 received in step S26-2 to the server 50, and the transmitter / receiver 51 of the server 50 receives the identification information and the abnormality detection setting file of the photographing device 10 transmitted from the photographing device 10 (step S26-4). If the photographing device 10 is not powered on or the communication conditions are poor and communication between the server 50 and the photographing device 10 is not possible, the processes of steps S26-2, 3, and 4 are omitted.

[0190] The storage / reading unit 59 searches for and reads out the abnormality detection setting file corresponding to the identification information stored in the abnormality management DB 5001 (step S26-5).

[0191] If there is no corresponding anomaly detection setting file in the anomaly management DB 5001, an anomaly detection setting file corresponding to the identification information used in the search is created and stored in the anomaly management DB 5001. In this case, the settings in the anomaly detection setting file are set to predefined default settings.

[0192] The abnormality detection setting screen of FIG. 21 is displayed on the display 506, and the user selects settings for each item on the abnormality detection setting screen using the reception unit 52 to perform abnormality detection settings, receives the various set information, and creates an abnormality detection setting file common to the imaging device 10 and the server 50 (step S26-6).

[0193] The storage / readout unit 19 associates the identification information created in step S26-6 with the abnormality detection setting file and stores them in the abnormality management DB 5001 (step S25-7).

[0194] Next, the transmitting / receiving unit 11 of the imaging device 10 transmits the identification information and a request for an abnormality detection setting file of the server 50 to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the identification information and the request for an abnormality detection setting file of the server 50 transmitted from the imaging device 10 (step S26-8).The storage / reading unit 59 searches for and reads out the abnormality detection setting file corresponding to the identification information stored in the abnormality management DB 5001 from the identification information and the request for an abnormality detection setting file of the server 50 received in step S26-8 (step S26-9).

[0195] The transmitting / receiving unit 51 transmits the identification information and the anomaly detection setting file read in step S26-9 to the imaging device 10, and the transmitting / receiving unit 11 of the imaging device 10 receives the identification information and the anomaly detection setting file transmitted from the server 50 (step S26-10). The storage / reading unit 19 associates the identification information and the anomaly detection setting received in step S26-10 with each other and stores them in the anomaly management DB 1001 (step S26-11).

[0196] The reason why the photographing device 10 requests the abnormality detection setting file from the server 50 in step S26-8 is to take into consideration the possibility that the photographing device 10 is not turned on or the communication conditions are poor, making it impossible for the server 50 and the photographing device 10 to communicate.

[0197] Step S13 and subsequent steps are the same as those in the first embodiment, and therefore will not be described here.

[0198] ●Fourth abnormality detection process 30 and 31 are sequence diagrams showing the fourth abnormality detection process according to this embodiment. Here, an example is shown in which abnormality detection is set in the communication terminal 90, and abnormality detection is performed by the communication terminal 90 and the server 50. The user operates the reception unit 92 of the communication terminal 90 to enter a mode for making anomaly detection settings, and inputs identification information (user ID, model ID, TwinID, etc.) to receive the set information (step S27-1). The transmission / reception unit 91 of the communication terminal 90 transmits the identification information and a request for an anomaly detection setting file for the server 50 to the server 50, and the transmission / reception unit 51 of the server 50 receives the identification information and the request for an anomaly detection setting file for the server 50 transmitted from the communication terminal 90 (step S27-2). The storage / reading unit 59 searches for and reads out the anomaly detection setting file corresponding to the identification information stored in the anomaly management DB 5001, based on the identification information and the request for an anomaly detection setting file for the server 50 received in step S27-2 (step S27-3).

[0199] If there is no corresponding anomaly detection setting file in the anomaly management DB 5001, an anomaly detection setting file corresponding to the identification information used in the search is created and stored in the anomaly management DB 5001. In this case, the settings in the anomaly detection setting file are set to predetermined default settings.

[0200] The transceiver 51 of the server 50 transmits the identification information received in step S27-2 and the abnormality detection setting file of the server 50 to the communication terminal 90, and the transceiver 91 of the communication terminal 90 receives the abnormality detection setting file of the server 50 transmitted from the server 50 (step S27-4). The storage / reading unit 99 searches for and reads out the abnormality detection setting file corresponding to the identification information stored in the abnormality management DB 9001 from the identification information (step S27-5).

[0201] If there is no corresponding anomaly detection setting file in the anomaly management DB 9001, an anomaly detection setting file corresponding to the identification information used in the search is created and stored in the anomaly management DB 9001. In this case, the settings in the anomaly detection setting file are set to predetermined default settings.

[0202] The anomaly detection setting screen of Figure 21 is displayed on the display 906, and the user selects settings for each item on the anomaly detection setting screen in the reception unit 92 to perform anomaly detection settings, receives the various set information, and creates an anomaly detection setting file common to the communication terminal 90 and the server 50 (step S27-6).

[0203] The storage / readout unit 99 associates the identification information created in step S27-6 with the anomaly detection setting file and stores them in the anomaly management DB 9001 (step S27-7).

[0204] The transmitting / receiving unit 91 transmits the identification information and the abnormality detection setting file created in step S27-6 to the server 50, and the transmitting / receiving unit 51 of the server 50 receives the identification information and the abnormality detection setting file transmitted from the imaging device 10 (step S27-8). The storage / reading unit 59 associates the identification information and the abnormality detection setting received in step S27-8 with each other and stores them in the abnormality management DB 5001 (step S27-9).

[0205] <Detecting abnormalities in photography and communication terminals> When the user gives a shooting instruction from the shooting control unit 13, the shooting control unit 13 captures a still image or a video using the still image shooting unit 15 or the video shooting unit 14 (step S27-10), and the storage / reading unit 19 associates the shooting data indicating the captured still image or video with an image ID and stores it in the image management DB 1002 (step S27-11). The transmitter / receiver 11 of the photographing device 10 transmits the identification information, photographed data (still images, video, etc.) and an abnormality detection file to the communication terminal 90 when triggered by a predetermined condition, and the transmitter / receiver 91 of the communication terminal 90 receives the identification information and photographed data transmitted from the photographing device 10 (step S27-12). Here, the predetermined condition that serves as a trigger is when photographing by the photographing device 10 ends or when the photographing device 10 becomes able to connect (synchronize) with the communication terminal 90 via the communication network 100. The memory / readout unit 99 stores the photographed data (still images, video, etc.) received in step S27-12 in the image management DB 9002 in association with the identification information (step S27-13).

[0206] The abnormality detection unit 96 performs abnormality detection processing on the communication terminal side for the still images or videos stored in the image management DB 9002 in step S27-13 in accordance with the abnormality detection settings (step S27-14), and the storage / readout unit 99 stores the information detected as an abnormality by the abnormality detection unit 66 in the abnormality management DB 9001 as an abnormality management file (step S27-15).

[0207] <Detection of abnormality in communication terminal 90> The detection by the communication terminal 90 in step S27-14 is suitable for processing that cannot be performed by the image capture device 10, processing that requires processing time, or processing that utilizes the functions of the communication terminal 90, as an intermediary between the image capture device 10 and the server 50. As an example of anomaly detection by the communication terminal 90, for example, the image capture data captured with the fisheye lens may be output and stored as is without stitching the image capture data in step S27-10, and the communication terminal 90 may perform the aforementioned one-sided blur detection on the image capture data captured with the fisheye lens. Alternatively, the image data after stitching processing by the communication terminal 90 may be sent to the server 50. Performing anomaly detection processing by the image capture device 10 may increase processing time, which may cause delays in the image capture sequence. Performing anomaly detection by the communication terminal 90 allows anomaly detection to be performed without affecting the image capture sequence.

[0208] The transmitting / receiving unit 91 of the communication terminal 90 transmits the identification information, the photographed data (still images, video, etc.) and the anomaly detection file to the server 50 when triggered by a predetermined condition, and the transmitting / receiving unit 51 of the server 50 receives the identification information, the photographed data and the anomaly detection file transmitted from the photographing device 10 (step S27-16). Here, the predetermined condition that serves as the trigger is when the communication terminal 90 becomes able to connect (synchronize) with the server 50 via the communication network 100. The storage / reading unit 59 stores the photographed data (still images, video, etc.) received in step S27-16 in the image management DB 5002 in association with the identification information (step S27-17).

[0209] The storage / reading unit 59 searches for and reads out the anomaly detection file corresponding to the identification information stored in the anomaly management DB 5001 from the identification information received in step S27-17, compares the anomaly detection file of the communication terminal 90 received in step S27-17 with the anomaly detection file of the server 50, and if there is a difference, generates an anomaly detection file by complementing the two files and stores it in the anomaly management DB 5001 (step S27-18).

[0210] If there is a difference between the anomaly detection files of the communication terminal 90 and the server 50 in step S27-18 and a supplementary anomaly detection file is created, the transmission / reception unit 51 transmits the identification information and the anomaly detection file created in step S27-18 to the communication terminal 90, and the transmission / reception unit 91 of the communication terminal 90 receives the identification information and the anomaly detection file transmitted from the server 50 (step S27-19).

[0211] The abnormality detection unit 96 checks the abnormality detection file received in step S27-19, and if an abnormality error code indicating an abnormality detected by the server 50 is recorded, the storage and reading unit 99 reads the abnormality detection setting set in step S27-6 and stored in the abnormality management DB 9001 in step S27-7, and if an error notification setting is set in the communication terminal 90, displays the abnormality detection information (error content, error code, etc.) on the display 906 of the communication terminal 90 (step S27-20). The storage and reading unit 99 associates the identification information received in step S27-20 with the abnormality detection file and stores them in the abnormality management DB 9001 (step S27-21).

[0212] Step S23 and subsequent steps are the same as those in the first embodiment, and therefore will not be described here.

[0213] [Major Effects of the Embodiments] As described above, detection of an abnormality occurring in the image capturing device 10 can be performed using an appropriate device and an appropriate method.

[0214] 〔others〕 The imaging device 10 or the communication terminal 90, which is an example of a source device, and the server 50, which is an example of a destination device, are configured to be usable by, for example, real estate agents who manage or sell real estate properties, real estate agents who introduce real estate properties, and construction companies that manage structures such as buildings, but the application of the present invention is not limited to these.

[0215] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in the present embodiment includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices designed to perform each function described above, such as an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a system on a chip (SOC), a graphics processing unit (GPU), and a conventional circuit module.

[0216] So far, we have described an information processing device, information processing method, program, and information processing system according to one embodiment of the present invention, but the present invention is not limited to the above-described embodiment, and can be modified within the scope of what a person skilled in the art can conceive, such as adding, changing, or deleting other embodiments, and as long as the functions and effects of the present invention are achieved in any aspect, it is included in the scope of the present invention. [Explanation of symbols]

[0217] 1. Image processing system (an example of an information processing system, an example of a system) 10. Imaging device (example of transmitting device) 11 Transmitting / receiving unit (an example of a transmitting means or a receiving means) 12 Operation reception section 13. Imaging control unit (an example of imaging control means) 14 Videography Department 15 Still Image Shooting Section 16 Abnormality detection unit (an example of a detection means) 19 Memory / readout unit (an example of setting means) 20 Support member 50 Server (an example of an information processing device or a destination device) 51 Transmitting / receiving unit (an example of receiving means, transmitting means, and notifying means) 56 Abnormality detection unit (an example of a detection means) 58 Path generation unit (an example of path generation means) 59 Memory / readout unit (an example of setting means) 90 Communication terminal (an example of an information processing device, a source device) 91 Transmitting / receiving unit (an example of a transmitting means) 92 Reception unit (an example of a setting means) 93 Display control unit (an example of display control means) 96 Abnormality detection unit (an example of detection means) 906 Display (Example of display unit) 1000 Memory unit (an example of memory means) 5000 Memory unit (an example of a memory means) 9000 Memory unit (an example of memory means)

Claims

1. An information processing system having a photographing device that captures photographed images by photographing, and an information processing device that can communicate with the photographing device, a storage means for storing setting information in which defects in the captured image and notification destinations for the defects are associated with each other; a detection means for detecting defects in the captured image; a notification means for notifying the notification destination that the defect has been detected based on the setting information; An information processing system having the above.

2. The setting information includes a type of the defect and a predetermined notification destination corresponding to the type, The notification means notifies the predetermined notification destination that the defect has been detected based on the setting information. The information processing system according to claim 1 .

3. the setting information includes a predetermined device among the photographing device and the information processing device that detects the defect and the predetermined notification destination corresponding to the predetermined device; The notification means notifies the predetermined notification destination that the defect has been detected based on the setting information. The information processing system according to claim 1 .

4. 2. The information processing system according to claim 1, The setting information includes details of the defect and whether or not the captured image can be corrected according to the details. an information processing system having a correction means for performing the correction based on the setting information;

5. 2. The information processing system according to claim 1, wherein if the defect is detected after a predetermined time has elapsed, the notification means in the photographing device notifies the notification destination that the defect has been detected based on the setting information.

6. the photographing device has a first detection means for detecting whether or not there is a defect in the photographed image; 2. The information processing system according to claim 1, wherein said information processing device comprises second detection means for detecting a faulty part of said photographing device that is the source of the defect in said photographed image.

7. 1. An information processing method executed by an information processing system having a photographing device that captures photographed images by photographing, and an information processing device that can communicate with the photographing device, the information processing system includes a storage means for storing setting information in which defects in the captured images and notification destinations for the defects are associated with each other, The information processing system includes: a detection process for detecting defects in the captured image; a notification process for notifying the notification destination that the defect has been detected based on the setting information; An information processing method that performs the above.

Images