Imaging system

Abstract

To allow images to be processed flexibly.SOLUTION: An imaging system includes a first image acquisition unit that acquires an image, a second image acquisition unit that acquires an image, a communication unit that transmits first information regarding a relative positional relationship between the first image acquisition unit and the second image acquisition unit to an external device, and a processing unit that performs processing determined by the external device based on the first information, on first data output from the first image acquisition unit.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an imaging system.

Background Art

[0002] An image processing apparatus that synthesizes images captured by surveillance cameras to generate a synthesized image is known (see Patent Document 1). In the prior art, it has been difficult for cameras and image processing apparatuses to perform flexible processing on images.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] An imaging system according to a first aspect of the present invention includes a first image acquisition unit that acquires an image, a second image acquisition unit that acquires an image, a communication unit that transmits first information regarding a relative positional relationship between the first image acquisition unit and the second image acquisition unit to an external device, and a processing unit that performs processing determined by the external device based on the first information on first data output from the first image acquisition unit.

Brief Description of the Drawings

[0005]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0006] Hereinafter, embodiments of the invention will be described with reference to the drawings. (First Embodiment) <Overview of the Imaging System> FIG. 1 is a schematic diagram illustrating an imaging system 1 according to the first embodiment of the invention. The imaging system 1 is composed of a plurality of camera units 10 and a plurality of servers 50A and 50B called a cloud 50. As the camera units 10, camera units 10A, 10B, 10C, 10D, and 10E are illustrated. Each camera unit 10 is connected to the cloud 50 via a wired or wireless network communication network. FIG. 1 illustrates the case of wireless connection. The number of camera units 10 may be increased or decreased without being limited to the five shown in the figure, and the number of servers constituting the cloud 50 may also be increased or decreased without being limited to the two shown in the figure.

[0007] Note that connecting the camera unit 10 and the cloud 50 via a network communication network is just an example, and for example, it may be configured as a connection in a local network that is not connected to the Internet, or as a connection within a device that is not connected to the local network. When configured as a connection within a device, the camera unit 10 is connected to an image processing unit (which may also be referred to as an image processing device) instead of the cloud 50.

[0008] Each camera unit 10 has at least one imaging lens and an imaging sensor, and can acquire one or more images for each camera unit 10. In the example of FIG. 1, the camera unit 10 is distinguished when at least one of the number of imaging lenses of one camera unit 10, the optical characteristics of the optical members including the imaging lens, the presence or absence of a sound collecting microphone in the camera unit 10, etc. is different, and different symbols (10A, 10B, 10C, 10D, and 10E) are attached.

[0009] For example, the camera unit 10A has one imaging unit and acquires one image. The camera units 10B, 10D, and 10E have a first imaging unit and a second imaging unit, and acquire two images. The camera unit 10E incorporates sound collecting microphones M1 and M2 for acquiring sound, while the camera unit 10B does not incorporate a sound collecting microphone. Also, the camera unit 10C has a first imaging unit, a second imaging unit, a third imaging unit, and a fourth imaging unit, and acquires four images.

[0010] The image data acquired by the camera units 10A, 10B, 10C, 10D, and 10E are each transmitted to the cloud 50 and can be stored in the server 50A or 50B.

[0011] The image acquired by the camera unit 10 may be a still image or a moving image. Also, the image may be a planar image called 2D or a stereoscopic image (obtaining a sense of depth based on parallax) called 3D. The 2D image includes a so-called omnidirectional image called VR360. Also, the 3D image includes a stereoscopic image using parallax such as VR180. The VR180 image may or may not conform to the format of live-action VR (Virtual Reality) with a 180-degree field of view in all directions proposed by Google (trademark). The VR180 image can be acquired, for example, by a camera unit in which two fisheye lenses are arranged side by side as the photographing lens as shown in the camera unit 10D.

[0012] User 2 can view, for example, a 2D image stored in the cloud 50 using a terminal 200 such as a smartphone. User 2 who has been registered in the cloud 50 in advance connects the terminal 200 to the cloud 50 via a network communication network and views any image stored in the cloud 50. The cloud 50 reproduces the data of the specified image and distributes it to the terminal 200 of User 2. Note that using the cloud 50 as the storage destination of the image is just an example, and the image may be stored in a camera unit other than the cloud 50, a station server described later, or the terminal 200.

[0013] User 2 can also view, for example, a 3D image stored in the cloud 50 using a head-mounted display 100 or the like. When User 2 connects to the cloud 50 via a network communication network, the cloud 50 reproduces the data of the specified image and distributes it to the head-mounted display 100 of User 2.

[0014] The images stored in the server 50A or 50B of the cloud 50 may be used for monitoring purposes. For example, a monitoring device connected to the cloud 50 monitors the presence or absence of abnormalities based on the images acquired by each camera unit 10 and transmitted to the cloud 50.

[0015] In addition, an intermediate server 40 (which may also be referred to as a station server) may be provided between the camera unit 10 and the server of the cloud 50. FIG. 2 is a schematic diagram illustrating an imaging system 1A provided with the station server 40. The imaging system 1A is composed of six camera units B (B-1 to B-6) corresponding to the camera unit 10D in FIG. 1, a station server 40, and a cloud 50. The station server 40 is composed of, for example, three servers S-1 to S-3. The cloud 50 is composed of, for example, one server C-1. The number of camera units B is not limited to the six shown in the figure and may be increased or decreased. The number of servers constituting the station server 40 is also not limited to the three shown in the figure and may be increased or decreased. The same applies to the number of servers constituting the cloud 50.

[0016] The camera units B-1 to B-3 are respectively connected to the servers S-1 and S-2 of the station server 40. The servers S-1 and S-2 are interconnected within the station server 40. In addition, the camera units B-4 to B-6 are respectively connected to the server S-3 of the station server 40.

[0017] In addition, the arrows indicated by broken lines between the server S-2 and the server S-3, between the camera unit B-4 and the server S-2, between the camera unit B-3 and the server S-3, and between the camera unit B-3 and the camera unit B-4 indicate, for example, paths that can be connected in the event of an emergency such as when the server S-1 fails. When the server S-1 is inoperable, the imaging system 1A can be operated in the same manner as in normal times by connecting the emergency connection paths.

[0018] In the imaging system 1 of FIG. 1, the first-stage signal processing for the images acquired by the camera units 10A, 10B, 10C, 10D, and 10E and the second-stage signal processing for the images after the first-stage signal processing are performed mainly by the camera unit 10 that acquired the images, or mainly by the cloud 50 (servers 50A and 50B). In other words, the first-stage and second-stage signal processing can be appropriately shared between the camera units 10A, 10B, 10C, 10D, and 10E and the cloud 50 (servers 50A and 50B). The sharing includes the case where the camera unit 10 is responsible for all of the first-stage and second-stage signal processing and the case where the signal processing is shared between the camera unit 10 and the cloud 50. Also, in the imaging system 1A of FIG. 2, the first-stage signal processing for the images acquired by the camera units B-1, B-2, B-3, B-4, B-5, and B-6 and the second-stage signal processing for the images after the first-stage signal processing are configured to be appropriately shared between the camera unit B-n (n is 1 to 6) that acquired the images, the station server 40 (servers S-1 to S-3), and the cloud 50 (server C-1). The sharing includes the case where the camera unit B-n and the station server 40 share the first-stage and second-stage signal processing and the case where the camera unit B-n, the station server 40, and the cloud 50 share the first-stage and second-stage signal processing. The configurations of the servers S-1 to S-3 of the station server 40 are the same as the configurations of the servers 50A (50B) of the cloud 50, which will be described in detail later. Also, the configuration of the server C-1 is the same as the configuration of the servers 50A (50B). In the following description, the case where the first-stage signal processing and the second-stage signal processing are performed as the signal processing for the images is exemplified. However, the signal processing may be further divided into a plurality of stages, and each signal processing may be appropriately shared between the camera unit 10 and the cloud 50. For example, the signal processing divided into the first stage to the fourth stage is shared by the camera unit 10, the cloud 50, and the like. The imaging system 1 (imaging system 1A) will be described in more detail.

[0019] <Overview of the camera unit> FIG. 3 is a schematic diagram for explaining the overview of the camera unit 10. FIG. 3(a) is a perspective view illustrating the appearance of the camera unit 10A, FIG. 3(b) is a perspective view illustrating the appearance of the camera unit 10B, and FIG. 3(c) is a perspective view illustrating the appearance of the camera unit 10D.

[0020] (1) Overview of the camera unit 10A The camera unit 10A is an imaging device in which one imaging unit 10-1 is mounted on the base unit 10-2. The base unit 10-2 can attach one imaging unit 10-1. The imaging unit 10-1 includes a photographing lens, an imaging sensor, a diaphragm, a shutter mechanism, an optical zoom mechanism, a focus adjustment mechanism, a shake correction mechanism, an optical filter, etc. In the present embodiment, a case where a fisheye lens with a photographing angle of view of 190 degrees is used as the photographing lens of the camera unit 10A is exemplified. When a fisheye lens is used for the imaging unit 10-1 of the camera unit 10A, a wider range (190 degrees in the vertical and horizontal directions) than the hemispherical image having a field of view of 180 degrees in the vertical and horizontal directions can be photographed. Note that, instead of the fisheye lens, an imaging unit equipped with another photographing lens having a different use may be mounted. For example, an imaging unit equipped with a standard lens having a photographing angle of view of about 50 degrees, a wide-angle lens having a photographing angle of view of about 70 degrees, or a telephoto lens having a photographing angle of view of about 10 degrees can be mounted. A photographing lens with a narrow photographing angle of view is used for applications where a hemispherical image is not acquired.

[0021] (2) Overview of the camera unit 10B The camera unit 10B is an imaging device in which two imaging units 10-1a and 10-1b are mounted on a base portion 10-3. The base portion 10-3 can attach the two imaging units 10-1a and 10-1b. In the present embodiment, a case where a fisheye lens with a shooting angle of view of 190 degrees is used as the shooting lens of the camera unit 10B is exemplified. When fisheye lenses are used for the imaging units 10-1a and 10-1b of the camera unit 10B, it becomes possible to obtain the above-described VR360 omnidirectional (so-called full-sphere) image by joining two hemispherical images. Note that instead of the fisheye lens, an imaging unit equipped with another shooting lens having a different use may be mounted. For example, an imaging unit equipped with a standard lens with a shooting angle of view of about 50 degrees, a wide-angle lens with a shooting angle of view of about 70 degrees, or a telephoto lens with a shooting angle of view of about 10 degrees can be mounted. A shooting lens with a narrow shooting angle of view is used for applications where a VR360 image is not obtained.

[0022] (3) Overview of the camera unit 10D The camera unit 10D is an imaging device in which two imaging units 10-1a and 10-1b are mounted on a base portion 10-4. The base portion 10-4 can attach the two imaging units 10-1a and 10-1b. In the present embodiment, a case where a fisheye lens with a shooting angle of view of 190 degrees is used as the shooting lens of the camera unit 10D is exemplified. When fisheye lenses are used for the imaging units 10-1a and 10-1b of the camera unit 10D, it becomes possible to obtain the above-described VR180 image based on two hemispherical images. Note that instead of the fisheye lens, an imaging unit equipped with another shooting lens having a different use may be mounted. For example, an imaging unit equipped with a standard lens with a shooting angle of view of about 50 degrees, a wide-angle lens with a shooting angle of view of about 70 degrees, or a telephoto lens with a shooting angle of view of about 10 degrees can be mounted. A shooting lens with a narrow shooting angle of view is used for applications where a hemispherical image is not obtained.

[0023] The camera units 10B and 10D are common in that both have two imaging units 10-1a and 10-1b. However, they differ in that the orientations of the two mounted imaging units 10-1a and 10-1b are different between the base unit 10-3 and the base unit 10-4. That is, in the camera unit 10B, the two imaging units 10-1a and 10-1b capture different subjects located in opposite directions, while in the camera unit 10D, the two imaging units 10-1a and 10-1b capture a common subject located in the same direction.

[0024] (4) Overview of Camera Units 10C and 10E Perspective views of the camera unit 10C and the camera unit 10E are not shown, but each has an appearance as illustrated in FIG. 1. The camera units 10A, 10B, 10C, 10D, and 10E are installed in all outdoor and indoor locations, for example, for purposes such as monitoring, observation, and appreciation. Outdoors, they can be installed on poles or on the outer walls of buildings, etc. Indoors, they can be installed on ceilings, walls, and pillars, etc. Also, the camera unit 10 may be mounted on a flying drone or the like, and furthermore, it may be installed inside the cabins of aircraft, ships, trains, vehicles, etc.

[0025] There are two installation methods: one is to add each camera unit 10, and the other is to embed the base part of the camera unit 10 in the ceiling, wall, etc. in advance, and then attach the imaging unit 10-1 to the base part as needed. Also, it is possible to remove the imaging unit 10-1 of the already installed camera unit 10 and replace it with another imaging unit 10-1 with the same specifications or another imaging unit 10-1 with different specifications. The imaging units 10 (10-1a, 10-1b, …) are commonly configured with a mount part for attaching to the base part, and can be easily attached to any base part.

[0026] <Configuration of Camera Unit> (1) Configuration of Camera Unit 10A The configuration of the camera unit 10A will be described with reference to the block diagram illustrated in FIG. 4. The camera unit 10A is composed of an imaging unit 10-1 and a base unit 10-2. The imaging unit 10-1 includes a photographing lens 210, an imaging sensor 220, and an aperture, a shutter mechanism, etc. (not shown above). Further, the base unit 10-2 includes a control unit 510, a first processing unit 520, a second processing unit 530, a position detection unit 540, a communication unit 550, and a power supply unit 560.

[0027] As described above, a fisheye lens with a photographing angle of view of 190 degrees is used as the photographing lens 210. Therefore, the imaging unit 10-1 can photograph a range wider than a hemisphere centered on the optical axis X200. Note that, among the directions indicated by the optical axis X200 of the photographing lens 210, the subject side will be referred to as the direction of the imaging unit 10-1.

[0028] The control unit 510 is composed of a CPU, a ROM, a RAM, etc., and controls the operations of each part of the imaging unit 10-1 based on a control program. The control unit 510 causes the imaging unit 10-1 to perform an imaging operation based on the instructions of the control program. The first processing unit 520 performs first-stage signal processing on the image signal output from the imaging sensor 220 of the imaging unit 10-1. In the present embodiment, as an example of the first-stage signal processing, the first processing unit 520 performs color tone adjustment processing, image quality adjustment processing, resolution adjustment processing, and compression processing when acquiring a moving image. The data of the image compressed by the compression processing can reduce the communication volume when transmitting to the cloud 50 compared with the non-compressed data.

[0029] The second processing unit 530 performs second-stage signal processing on the signal of the image after the first-stage signal processing. In the present embodiment, as an example of the second-stage signal processing, the second processing unit 530 performs subject extraction processing, subject recognition processing, face recognition processing, and processing for improving appearance (for example, beauty processing for portrait photos), etc. Further, as the second-stage signal processing, the second processing unit 530 can also perform conversion processing for converting the hemispherical image acquired via the fish-eye lens into a two-dimensional image, for example, an image in the equirectangular projection method. Note that when there is no plan to perform the second-stage signal processing, etc., the second processing unit 530 may be omitted.

[0030] The position detection unit 540 detects the positional relationship between the two imaging units 10-1 when two camera units 10A are used in combination. Specifically, it is possible to detect geometric information indicating the distance and direction between the two imaging units 10-1, inclination information of each imaging unit 10-1 with respect to the ground, and the like. Based on the detected geometric information and inclination information between the two imaging units 10-1, for example, it is possible to accurately detect whether the directions of the two imaging units 10-1 are the same or opposite. The position detection unit 540 may also perform position detection of the base unit 10-2 itself. Note that when there is no plan to use two camera units 10A in combination, the position detection unit 540 of the base unit 10-2 may be omitted.

[0031] The communication unit 550 communicates with the cloud 50 or the station server 40. The power supply unit 560 supplies necessary power to each part of the base unit 10-2 and the imaging unit 10-1 mounted on the base unit 10-2. The power supply unit 560 may include a battery 570 as its power source.

[0032] (2) Configuration of the camera unit 10D The configuration of the above-described camera unit 10D will be described with reference to the block diagram illustrated in FIG. 5. The camera unit 10D is composed of two imaging units 10-1a and 10-1b, and a base unit 10-4. The first imaging unit 10-1a includes a photographing lens 210, an imaging sensor 220, and an aperture, a shutter mechanism, etc. (not shown above). The second imaging unit 10-1b includes a photographing lens 310, an imaging sensor 320, and an aperture, a shutter mechanism, etc. (not shown above). As described above, by using a fisheye lens with a photographing angle of view of 190 degrees for the imaging unit 10-1a, it is possible to photograph a range wider than a hemisphere centered on the optical axis X200. Similarly, by using a fisheye lens with a photographing angle of view of 190 degrees for the imaging unit 10-1b, it is possible to photograph a range wider than a hemisphere centered on the optical axis X300. The base unit 10-4 includes a control unit 510, a first processing unit 520, a second processing unit 530, a position detection unit 540, a communication unit 550, and a power supply unit 560.

[0033] The configuration of each part in the base unit 10-4 of the camera unit 10D is the same as that of the base unit 10-2 of the camera unit 10A. However, the control unit 510 of the base unit 10-4 controls the imaging of both the first imaging unit 10-1a and the second imaging unit 10-1b based on the instructions of the control program, which is different from the control unit 510 of the base unit 10-2 that controls the imaging of one imaging unit 10-1. In addition, the first processing unit 520 of the base unit 10-4 performs the first-stage signal processing on the image signal output from the imaging sensor 220 of the first imaging unit 10-1a and the first-stage signal processing on the image signal output from the imaging sensor 320 of the second imaging unit 10-1b, which is also different from the first processing unit 520 of the base unit 10-2 that performs the first-stage signal processing on the image signal output from the imaging sensor 220 of one imaging unit 10-1.

[0034] As an example of the first-stage signal processing, the first processing unit 520 of the camera unit 10D performs color adjustment processing, image quality adjustment processing, resolution adjustment processing, and compression processing when acquiring a moving image, which are the same as those of the camera unit 10A. Also, when it is necessary to align the color, image quality, number of pixels, size, etc. between the image acquired by the first imaging unit 10-1a and the image acquired by the second imaging unit 10-1b, color adjustment processing, image quality adjustment processing, resolution adjustment processing, etc. between the two images may be performed as the first-stage signal processing.

[0035] In this embodiment, the first-stage signal processing is referred to as the first processing α. When the content or combination of signal processing and adjustment processing performed as the first-stage signal processing differs between camera units, it may be referred to as the first processing α-1, the first processing α-2, ….

[0036] As an example of the second-stage signal processing, the second processing unit 530 of the camera unit 10D performs subject extraction processing, subject recognition processing, face recognition processing, processing to improve appearance, and conversion processing, which are the same as those of the camera unit 10A. Also, as the second-stage signal processing, the second processing unit 530 may perform processing to generate a display image for viewing by the user 2 using the terminal 200, the head-mounted display 100, etc., based on at least one of the data of the first and second images acquired by the first imaging unit 10-1a and the second imaging unit 10-1b, for example.

[0037] The second processing unit 530 of the camera unit 10D further performs, as the second-stage signal processing, shake reduction processing called electronic gimbal processing, lens distortion correction processing to correct lens distortion caused by the optical characteristics of the photographing lens 210 and the photographing lens 310, depth calculation processing to calculate the distance (i.e.,) from the camera unit 10D to the object, and extraction processing to extract the main subject by background blurring, etc. In this embodiment, the signal processing in the second stage is referred to as second process β. When the content or combination of each process performed as the signal processing in the second stage differs between camera units, it may be referred to as second process β-1, second process β-2, ….

[0038] The depth calculation will be briefly described. In camera unit 10D, a stereo camera is configured in which the optical axis X200 of the first imaging unit 10-1a and the optical axis X300 of the second imaging unit 10-1b are arranged with a distance L (referred to as the baseline length) therebetween. The photographing lenses 210 and 310 are constituted by, for example, equidistant projection type fish-eye lenses. With such a configuration, on the images obtained by the imaging elements 220 and 320, the distances from the respective optical axes X200 and X300 represent the direction of the object (subject). Also, the difference in the positions on the images of the same object imaged on the images obtained by the imaging elements 220 and 320 represents the parallax. Therefore, based on the image by the imaging element 220 and the image by the imaging element 320, the distance to the object can be obtained using the principle of triangulation.

[0039] The position detection unit 540 of the camera unit 10D detects the positional relationship between the first imaging unit 10-1a and the second imaging unit 10-1b. Specifically, it is possible to detect the geometric information between the first imaging unit 10-1a and the second imaging unit 10-1b, the inclination information of both imaging units 10-1a and 10-1b with respect to the ground, and the like. In the camera unit 10D, originally, the optical axis X200 of the first imaging unit 10-1a and the optical axis X300 of the second imaging unit 10-1b are designed to be substantially parallel. Also, the baseline length L is designed to be, for example, 65 mm. Therefore, the position detection unit 540 of the camera unit 10D detects that the optical axes X200 and X300 are parallel and that the baseline length L is 65 mm. By setting the baseline length L to 65 mm, which is the same as the distance between human eyes, a stereo-form image with a sense of depth similar to that seen by human eyes can be obtained.

[0040] Note that the geometric information between the first imaging unit 10-1a and the second imaging unit 10-1b can be detected using the distance and direction detection function between sensors or wireless communication devices such as Bluetooth (trademark). Further, the inclination information can be detected using an inclination sensor or the like. Regarding the geometric information, instead of using sensors or wireless communication devices, the following method of recording the geometric information in advance may be adopted. That is, based on the design information of the attachment portions of the first imaging unit 10-1a and the second imaging unit 10-1b in the base portion 10-4, information such as the interval between the optical axis X200 and the optical axis X300 (= baseline length L) is recorded in advance in the ROM of the control unit 510. The position detection unit 540 obtains the geometric information between the first imaging unit 10-1a and the second imaging unit 10-1b by reading the information recorded in the control unit 510. The position detection unit 540 may perform position detection of the base portion 10-4 itself.

[0041] The communication unit 550 of the camera unit 10D communicates with the cloud 50 or the station server 40. The power supply unit 560 of the camera unit 10D supplies necessary power to each part of the base portion 10-4 and the first imaging unit 10-1a and the second imaging unit 10-1b mounted on the base portion 10-4.

[0042] (3) Configuration of Camera Units 10B and 10E The camera units 10B and 10E differ from the case of the camera unit 10D in that the orientations of the two imaging units 10-1a and 10-1b are different and the information on the baseline length L is not required. However, the configurations of the imaging units 10-1a and 10-1b shown in the block diagram are common to the case of the camera unit 10D (FIG. 5). Therefore, the configurations of the camera units 10B and 10E are not shown and will be described with reference to FIG. 5.

[0043] In camera units 10B and 10E, the first imaging unit 10-1a includes a photographing lens 210 and an imaging sensor 220. The second imaging unit 10-1b includes a photographing lens 310 and an imaging sensor 320. As described above, the imaging unit 10-1a of the camera units 10B and 10E can photograph a range wider than a hemisphere centered on the optical axis X200 by using a fisheye lens with a photographing angle of view of 190 degrees. Similarly, the imaging unit 10-1b can photograph a range wider than a hemisphere centered on the optical axis X300 by using a fisheye lens with a photographing angle of view of 190 degrees. Furthermore, the configurations of the respective parts in the base unit 10-3 of the camera unit 10B and the configurations of the respective parts in the base unit (not shown) of the camera unit 10E are the same as the configurations of the respective parts in the base unit 10-4 of the camera unit 10D.

[0044] In the camera units 10B and 10E, the control unit 510 controls the imaging of both the first imaging unit 10-1a and the second imaging unit 10-1b based on the instructions of the control program, and the first processing unit 520 performs the first-stage signal processing on the image signal output from the imaging sensor 220 of the first imaging unit 10-1a and the first-stage signal processing on the image signal output from the imaging sensor 320 of the second imaging unit 10-1b, which is the same as the case of the camera unit 10D.

[0045] The camera units 10B and 10E are different from the camera unit 10D in the following points. That is, in addition to the second-stage signal processing exemplified in the description of the camera unit 10A, the second processing unit 530 performs, as the second-stage signal processing, a combining process (referred to as a stitch process) of joining two hemispherical images based on the data of the first and second images acquired by the first imaging unit 10-1a and the second imaging unit 10-1b to obtain the above-described VR360 omnidirectional (full spherical) image. Here, since the photographing lenses 210 and 310 have an angle of view of 190 degrees, the imaging unit 10-1a photographs a range wider than a hemisphere centered on the optical axis X200, and the imaging unit 10-1b photographs a range wider than a hemisphere centered on the optical axis X300. Therefore, since so-called overlap is secured in the stitching process of joining two hemispherical images, it becomes possible to accurately perform image joining.

[0046] Note that the second processing unit 530 of the camera units 10B and 10E performs, as the second-stage signal processing, a shake reduction process called an electronic gimbal process and a lens distortion correction process for correcting lens distortion caused by the optical characteristics of the photographing lenses 210 and 310, etc., which is the same as in the case of the camera unit 10D. As described above, in the present embodiment, the second-stage signal processing is referred to as the second process β. However, when the content of each process or the combination of processes performed as the second-stage signal processing differs between camera units, it may be referred to as the second process β-1, the second process β-2, ….

[0047] <Combination of a plurality of camera units 10A> In the present embodiment, by combining a plurality of camera units 10A, they can be treated in the same manner as the camera units 10B, 10E, or 10D. FIG. 6 is a diagram showing an arrangement example of nine camera units 10A (10A-1 to 10A-9). Among the nine camera units 10A, the camera units 10A-1 and 10A-2 that can photograph different subjects located in opposite directions can be treated in the same manner as the camera unit 10B (10E). Also, among the nine camera units 10A, the camera units 10A-2 and 10A-3 that can photograph a common subject located in the same direction can be treated in the same manner as the camera unit 10D. Note that the arrangement example in FIG. 6 illustrates the case where the orientations of some imaging units of the camera units 10A (10A-1 to 10A-9) are the same (0 degrees) and the case where they are opposite (180 degrees). However, the arrangement may be such that the orientations of the imaging units differ by, for example, 90 degrees or, for example, 135 degrees.

[0048] A configuration example of the combination of the camera unit 10A-2 and the camera unit 10A-3 will be described with reference to the block diagram of FIG. 7. The first imaging unit 10-1a included in the camera unit 10A-2 includes a photographing lens 210 and an imaging sensor 220. The second imaging unit 10-1b included in the camera unit 10A-3 includes a photographing lens 310 and an imaging sensor 320. As described above, the first imaging unit 10-1a can photograph a range wider than a hemisphere centered on the optical axis X200. Also, the second imaging unit 10-1b can photograph a range wider than a hemisphere centered on the optical axis X300. The first base unit 10-2a included in the camera unit 10A-2 and the second base unit 10-2b included in the camera unit 10A-3 each include a control unit 510, a first processing unit 520, a second processing unit 530, a position detection unit 540, a communication unit 550, and a power supply unit 560.

[0049] In FIG. 7, the configurations of the control unit 510, the first processing unit 520, the second processing unit 530, the position detection unit 540, the communication unit 550, and the power supply unit 560 are the same as the configuration of the base unit 10-2 of the camera unit 10A described above. The position detection unit 540 when using two camera units 10A in combination can detect geometric information between the first imaging unit 10-1a included in the camera unit 10A-2 and the second imaging unit 10-1b included in the camera unit 10A-3, inclination information of both imaging units 10-1a, 10-1b with respect to the ground, etc. Based on the detected geometric information and inclination information between the two imaging units 10-1a, 10-1b, for example, it is possible to accurately detect whether the orientations of the first imaging unit 10-1a and the second imaging unit 10-1b are the same or opposite. In addition to the method of detecting geometric information between the first imaging unit 10-1a and the second imaging unit 10-1b using a sensor (not shown), etc., a method may be adopted in which geometric information is recorded in advance in the ROM in the control unit 510 and read out when necessary.

[0050] In particular, the first processing unit 520 in FIG. 7, in addition to the first-stage signal processing exemplified in the description of the camera unit 10A above, performs adjustment processing to align the hue, image quality, number of pixels, size, etc. between the image acquired by the first imaging unit 10-1a included in the camera unit 10A-2 and the image acquired by the second imaging unit 10-1b included in the camera unit 10A-3 as the first-stage signal processing.

[0051] In the present embodiment, the signal processing performed individually on the image acquired by the first imaging unit 10-1a and the image acquired by the second imaging unit 10-1b, and the adjustment processing to align the hue, etc. between the image acquired by the first imaging unit 10-1a and the image acquired by the second imaging unit 10-1b are merely examples of the first-stage signal processing. Also, the first-stage signal processing may be performed by either one of the first processing unit 520 included in the camera unit 10A-2 and the first processing unit 520 included in the camera unit 10A-3, or may be shared and processed by both.

[0052] In addition to the second-stage signal processing exemplified in the description of the camera unit 10A, the second processing unit 530 performs processing to generate a display image for viewing by the user 2 using the terminal 200, the head-mounted display 100, etc. based on at least one of the data of the first and second images acquired by the first imaging unit 10-1a included in the camera unit 10A-2 and the second imaging unit 10-1b included in the camera unit 10A-3 as the second-stage signal processing.

[0053] Furthermore, the second processing unit 530 in FIG. 7 may perform, as the second-stage signal processing, a shake reduction process called electronic gimbal processing, a lens distortion correction process for correcting lens distortion caused by the optical characteristics of the photographing lenses 210 and 310, a depth calculation process for calculating the distance (i.e., depth) from the camera units 10A-2 and 10A-3 to the object, an extraction process for extracting the main subject by background blur, and the like. The second-stage signal processing may be shared and processed by either one or both of the second processing unit 530 included in the camera unit 10A-2 and the second processing unit 530 included in the camera unit 10A-3.

[0054] The communication units 550 of both the camera unit 10A-2 and the camera unit 10A-3 perform communication between the camera unit 10A-2 and the camera unit 10A-3. In addition, one of the communication units 550 of the camera unit 10A-2 and the camera unit 10A-3 communicates with the cloud 50 or the station server 40 on behalf of the camera unit 10A-2 and the camera unit 10A-3.

[0055] <Configuration of the cloud> (1) Server 50A (50B) The configuration of the server 50A (50B) of the cloud 50 will be described with reference to the block diagram illustrated in FIG. 8. The server 50A (50B) includes a control unit 551, a first processing unit 552, a second processing unit 553, a face detection unit 554, a face recognition unit 555, a communication unit 556, a power supply unit 557, and a storage unit 558.

[0056] The control unit 551 is composed of a CPU, a ROM, a RAM, etc., and controls the operations of each part of the server 50A (50B) based on a control program. In addition, the control unit 551 determines the entity responsible for processing, such as whether the first-stage signal processing for the image signal acquired by the camera unit 10 and the second-stage signal processing for the image after the first-stage signal processing are performed by the camera unit 10, the station server 40, the cloud 50, or shared. The determination of the processing entity responsible for the processing is not limited to the server 50A (50B) of the cloud 50, and it may be determined by the camera unit 10 or the station server 40. In the following description, the case where the control unit 551 of the server 50A (50B) makes the determination will be exemplified.

[0057] The first processing unit 552 can perform the first-stage signal processing in the same manner as the first processing unit 520 of the camera unit 10. In the present embodiment, the first-stage signal processing for the image signal output from the imaging sensor 220 (or imaging sensor 320) of the camera unit 10 is usually performed by the first processing unit 520 in the camera unit 10 as the processing entity. However, for example, when the remaining amount of the battery 570 of the power supply unit 560 of the camera unit 10 becomes equal to or less than a predetermined value, or when the state of the processing load in the camera unit 10 is regarded as a problem (for example, the processing load is higher than a predetermined reference value), etc., the first processing unit 552 of the server 50A (50B) can perform the first-stage signal processing as the processing entity instead of the first processing unit 520 in the camera unit 10. The state of the processing load is determined, for example, by the occupancy rate of the processing in the CPU, etc. × the processing time, the frame rate (unit: fps) of the acquired moving image, etc. The larger these numerical values are, the higher the state of the processing load, and the smaller these numerical values are, the lower the state of the processing load. Also, even if the content of the signal processing is the same, when the processing ability of the CPU, etc. is high, the state of the processing load is low, and when the processing ability of the CPU, etc. is low, the state of the processing load is high. The processing ability and the state of the processing load of the CPU, etc. in the camera unit 10 are included in the information regarding the camera unit 10 described later and are transmitted from the camera unit 10 to the cloud 50.

[0058] The second processing unit 553 can perform second-stage signal processing in the same manner as the second processing unit 530 of the camera unit 10. The second-stage signal processing may include conversion processing for converting the hemispherical image acquired via the fisheye lens into a two-dimensional image, for example, an image in the equirectangular projection method. Furthermore, the second-stage signal processing may include processing for adding production effects such as particle effects to the image.

[0059] In this embodiment, the second-stage signal processing for the signal of the image acquired by the imaging unit 10-1 (10-1a or 10-1b) of the camera unit 10 is usually performed by the second processing unit 553 in the server 50A (or 50B) as the main processing entity. However, for example, when there is a surplus in the remaining amount of the battery 570 of the power supply unit 560 of the camera unit 10 and the processing load state in the camera unit 10 is low, the second processing unit 530 in the camera unit 10 can perform the second-stage signal processing as the main processing entity instead of the second processing unit 553 of the server 50A (or 50B).

[0060] The face detection unit 554 detects the face of a human or an animal based on the image acquired by the imaging unit 10-1 (10-1a or 10-1b) of the camera unit 10. The face recognition unit 555 confirms that it is a specific person (for example, the person named ○○○○ himself) by comparing the face detected by the face detection unit 554 with the data registered (recorded) in the storage unit 558 in advance. In addition to the face recognition unit, an object recognition unit, an action recognition unit, an abnormality recognition unit, a scene recognition unit, etc. may be provided. Each recognition unit recognizes the object shown, the action of the object, the abnormality of the object, and the scene shown by comparing with the data registered (recorded) in the storage unit 558 in advance based on the acquired image.

[0061] The communication unit 556 communicates with the camera unit 10 or the station server 40. The power supply unit 557 supplies necessary power to each part of the server 50A (50B). The storage unit 558 stores data of images transmitted from the camera unit 10, data obtained by performing first-stage signal processing in the first processing unit 552, data obtained by performing second-stage signal processing in the second processing unit 553, and data registered for face recognition, etc. Recording and reading of data to and from the storage unit 558 are controlled by the control unit 551.

[0062] <Explanation of flowchart> In the imaging system 1 (1A) described above, the image acquired by the camera unit 10 is stored in the cloud 50 after predetermined processing is performed. As described above, in the present embodiment, the control unit 551 of the server 50A (50B) determines the entity that performs processing on the image. The determined processing entity can be further changed by the determination of the control unit 551 of the server 50A (50B) or the control unit 510 of the camera unit 10. Specifically, the first-stage signal processing on the signal of the image acquired by the camera unit 10 is performed by the first processing unit 552 of the server 50A (50B) as the processing entity instead of the first processing unit 520 in the camera unit 10, or the second-stage signal processing on the image after the above first-stage signal processing is performed by the second processing unit 530 in the camera unit 10 as the processing entity instead of the second processing unit 553 of the server 50A (50B). Note that the change of the above processing entity is not limited to the determination in the camera unit 10 or the server 50A (50B), and may be changed by the determination of the station server 40. Regarding the transfer of information and image data performed between the camera unit 10 and the cloud 50 in the imaging system 1, it will be described with reference to the flowcharts shown in FIGS. 9 and 10.

[0063] (Camera unit 10) FIG. 9 is a flowchart for explaining the flow of processing executed by the control unit 510 of the camera unit 10. The control unit 510 repeatedly executes the processing according to FIG. 9 while communication with the cloud 50 is established. Note that instead of repeatedly executing, it may be configured to execute when there is a trigger.

[0064] In step S100 of FIG. 9, the control unit 510 transmits information regarding the camera unit 10 from the communication unit 550 to the cloud 50 and proceeds to step S102. The information regarding the camera unit 10 includes optical characteristics of the photographing lens 210 (photographing lens 310) constituting the imaging unit 10-1 (focal length, angle of view, aperture F value, relative position from the attachment part to the base part to the photographing lens, distortion, aberration inherent in the optical system, type of control used in the imaging unit, presence or absence of a zoom mechanism, presence or absence of a diaphragm, information indicating the size of the camera unit 10, etc.), specifications of the imaging sensor 220 (imaging sensor 320) (size of the sensor, number of pixels, etc.), positional relationship between a plurality of imaging units 10-1, tilt information of both imaging units 10-1 with respect to the ground, etc., state of the processing load in the camera unit 10, state of heat generation of devices (imaging unit, base unit), and information indicating the remaining amount of the battery 570 in the power supply unit 560. Further, when the processing entity of the first-stage signal processing and the processing entity of the second-stage signal processing are determined by the camera unit 10, each determined processing entity is included in the information regarding the camera unit 10. Furthermore, when a storage unit is additionally provided in the camera unit 10 or the like, recordable information indicating that an image can be recorded in the camera unit 10 or non-recordable information indicating that an image cannot be recorded may be included. The non-recordable information indicates that the free capacity of the storage unit is less than a predetermined capacity or that there is a failure in the storage unit or the like. Note that, in the case of the camera unit 10A, since there is one imaging unit 10-1, the information indicating the positional relationship between a plurality of imaging units 10-1 may be omitted. Also, in the case of the camera unit 10C, since there are four imaging units 10-1, the information indicating the positional relationship between the four imaging units 10-1, tilt information of each imaging unit 10-1 with respect to the ground, etc. are included.

[0065] Among the information about the camera unit 10, the optical characteristics of the photographing lens 210 (photographing lens 310) and the specifications of the imaging sensor are required when the control unit 551 of the server 50A (50B) determines the content of the first process (α or α-1, α-2, …) as the first-stage signal processing, and in the determined process. Also, information indicating the positional relationship of a plurality of imaging units 10-1 is required when the control unit 551 of the server 50A (50B) determines the content of the second process (β or β-1, β-2, …) as the second-stage signal processing, and in the determined process. Furthermore, the remaining amount of the battery 570 in the power supply unit 560 and the state of the processing load in the camera unit 10 are required when the control unit 551 of the server 50A (50B) determines whether the execution subject of the first process and the second process, that is, whether to perform the process in the camera unit 10 or in the cloud 50. However, when the processing content of the first-stage signal processing and the processing content of the second-stage signal processing are determined on the camera unit 10 side, each processing content is determined using the information about the camera unit 10. The above information about the camera unit 10 is based on the combination of the base unit and the imaging unit that constitute the camera unit 10. For example, when the imaging unit with different specifications or the base unit with different specifications is replaced, the information about the camera unit 10 is determined by the specifications of the replaced imaging unit and base unit. Also, a camera unit in which the base unit and the imaging unit are integrally configured may be used. In the case of integral configuration, the information about the camera unit 10 is determined by the specifications of the camera unit after integral configuration.

[0066] In step S102, the control unit 510 determines whether it has received the processing content and the processing entity for the image from the cloud 50. As described above, in this embodiment, each processing entity of the first processing and the second processing and their respective processing contents are determined on the cloud 50 (server 50A (50B)) side. The processing content for the image is assumed to include the content of the first processing (α or α-1, α-2, …) as the first-stage signal processing, the content of the second processing (β or β-1, β-2, …) as the second-stage signal processing, and information indicating whether the first processing and the second processing are to be performed by the camera unit 10 or the cloud 50. When the control unit 510 receives the information indicating the processing content and the like from the cloud 50, it makes an affirmative determination in step S102 and proceeds to step S104. When it has not received the information indicating the processing content and the like, it makes a negative determination in step S102 and proceeds to step S110.

[0067] In step S104, the control unit 510 determines the processing content of the camera unit 10 for the image acquired by the imaging unit 10-1 based on the content of the information received from the cloud 50. For example, based on the information (instruction) received from the cloud 50, the base unit 10-2 of the camera unit 10 is determined to perform the first processing (α or α-1, α-2, …) as the processing entity, and it is determined to omit the processing by the camera unit 10 for the second processing (β or β-1, β-2, …), and then it proceeds to step S106.

[0068] In step S106, the control unit 510 determines whether the state of the camera unit 10 satisfies the condition for performing the processing on the image. It is assumed that the camera unit 10 has a condition set in advance that when the remaining amount of the battery 570 in the power supply unit 560 is, for example, 20% or more and the state of the processing load is below a predetermined threshold, the processing is to be performed by the camera unit 10. When the control unit 510 satisfies the above conditions, it makes an affirmative determination in step S106 and proceeds to step S108. When the above conditions are not satisfied, it makes a negative determination in step S106 and proceeds to step S112.

[0069] In step S108, when the imaging unit 10-1 of the camera unit 10 acquires an image, the control unit 510 causes the first processing unit 520 of the base unit 10-2, or the first processing unit 520 and the second processing unit 530 of the base unit 10-2, to perform the process determined in step S104 on the acquired image. After the processing, the image data is transmitted from the communication unit 550 to the cloud 50, and the process shown in FIG. 9 is terminated.

[0070] In step S110, which proceeds after the negative determination in step S102, the control unit 510 determines the predetermined processing content recorded in the control unit 510. For example, the first processing unit 520 of the base unit 10-2 of the camera unit 10 performs the first process α-1 as the processing entity, and determines to omit the processing of the second process β in the camera unit 10, and proceeds to step S106.

[0071] In step S112, which proceeds after the negative determination in step S106, the control unit 510 determines to omit the processing in the camera unit 10 for the first process (α or α-1, α-2, …) and the second process (β or β-1, β-2, …), and proceeds to step S114. Determining to omit the first process and the second process in the camera unit 10 is substantially equivalent to changing the processing entities of the first process and the second process from the camera unit 10 to the cloud 50.

[0072] In step S114, when the imaging unit 10-1 of the camera unit 10 acquires an image, the control unit 510 transmits the data of the unprocessed image from the communication unit 550 to the cloud 50 without performing processing on the acquired image by the first processing unit 520 and the second processing unit 530 of the base unit 10-2 of the camera unit 10, and the process shown in FIG. 9 is terminated.

[0073] (Cloud 50) FIG. 10 is a flowchart for explaining the flow of processing executed by the control unit 551 of the cloud 50 (server 50A (50B)). The control unit 551 repeatedly executes the processing according to FIG. 10 while communication is established with the camera unit 10. Therefore, when information is determined and changed on the camera unit 10 side, the latest information is transmitted to the cloud 50 in step S200 described later. Note that instead of repeatedly executing, it may be configured to execute when there is a trigger.

[0074] In step S200 of FIG. 10, the control unit 551 receives information regarding the camera unit 10 from the camera unit 10 and proceeds to step S202. In step S202, the control unit 551 determines the processing content and processing entity for the image based on the information regarding the camera unit 10 received in step S200. As described above, the determined content includes the content of the first processing (α or α-1, α-2,...) as the first-stage signal processing, the content of the second processing (β or β-1, β-2,...) as the second-stage signal processing, and information indicating whether the first processing and the second processing are to be performed by the camera unit 10 or the cloud 50. If the information regarding the camera unit 10 includes recordable information, the control unit 551 may determine the recording destination of the image in step S202. If the recording destination of the image is determined to be the camera unit 10, the image data is transmitted to the camera unit 10 for recording. If the recording destination of the image is determined to be the cloud 50, the image data is recorded in the storage unit 558. The following description exemplifies the case where the information regarding the camera unit 10 does not include recordable information (the recording destination of the image data is determined to be the storage unit 558).

[0075] There are roughly three patterns in the division of processing between the camera unit 10 and the cloud 50. The first pattern is a pattern in which the first processing (α or α-1, α-2,...) is performed by the camera unit 10 and the second processing (β or β-1, β-2,...) is performed by the cloud 50. The second is a pattern in which the camera unit 10 performs the first process (α or α-1, α-2, …) and the second process (β or β-1, β-2, …). The third is a pattern in which the camera unit 10 does not perform the process, and the cloud 50 performs the first process (α or α-1, α-2, …) and the second process (β or β-1, β-2, …).

[0076] In step S204, the control unit 551 transmits the processing content and the processing entity determined in step S202 from the communication unit 556 to the camera unit 10 and proceeds to step S206. When the cloud 50 side determines the processing content, the processing entity, etc. and changes the information, the latest information is transmitted to the camera unit 10 in step S204. In step S206, the control unit 551 determines whether or not it has received the image data from the camera unit 10. If the control unit 551 has received the image data from the camera unit 10, it makes an affirmative determination in step S206 and proceeds to step S208. If it has not received the image data, it makes a negative determination in step S206 and ends the process according to FIG. 10. Note that the image data received from the camera unit 10 is provided with information indicating whether or not the first process (α or α-1, α-2, …) as the first-stage signal processing and the second process (β or β-1, β-2, …) as the second-stage signal processing have been performed.

[0077] In step S208, the control unit 551 determines whether or not the first process (α or α-1, α-2, …) and the second process (β or β-1, β-2, …) have been performed on the image data. If the control unit 551 determines that the first process (α or α-1, α-2, …) and the second process (β or β-1, β-2, …) have been performed, it makes an affirmative determination in step S208 and proceeds to step S210. If the first process (α or α-1, α-2, …) and the second process (β or β-1, β-2, …) have not been performed, it makes a negative determination in step S208 and proceeds to step S212.

[0078] If proceeding to step S210, processing of the image at the cloud 50 is unnecessary. In step S210, the control unit 551 stores the received image in the storage unit 558 and ends the processing according to FIG. 10. If the negative determination is made in step S208, some processing of an image at the cloud 50 is necessary. In step S212, the control unit 551 determines whether a first process (α or α-1, α-2, …) has been performed on the image data. If the control unit 551 determines that the first process (α or α-1, α-2, …) has been performed, it makes an affirmative determination in step S212 and proceeds to step S214. If the first process (α or α-1, α-2, …) has not been performed, it makes a negative determination in step S212 and proceeds to step S216.

[0079] If proceeding to step S214, a second process (β or β-1, β-2, …) of the image at the cloud 50 is necessary. In step S214, the control unit 551 performs the second process (β or β-1, β-2, …) on the received image data using the second processing unit 553, stores the image data after the second process (β or β-1, β-2, …) in the storage unit 558, and ends the processing according to FIG. 10.

[0080] If the negative determination is made in step S212, both the first process (α or α-1, α-2, …) and the second process (β or β-1, β-2, …) of the image are necessary at the cloud 50. In step S216, the control unit 551 performs the first process (α or α-1, α-2, …) on the received image data using the first processing unit 552, further performs the second process (β or β-1, β-2, …) using the second processing unit 553, stores the image data after the first and second processes in the storage unit 558, and ends the processing according to FIG. 10.

[0081] <Determination of Image Processing> An example of the method by which the control unit 551 determines the first-stage signal processing (first process α, α-1, α-2, …) and the second-stage signal processing (second process β, β-1, β-2, …) among the processing contents of the image will be described with reference to FIG. 11. FIG. 11 is a table showing the entity that performs processing on an image and the first processing (α, α-1, α-2, …) and the second processing (β, β-1, β-2, …) among the processing contents when the camera unit 10 has two imaging units 10. In this table, based on the combination of the two imaging units 10, the entity that performs processing on the image and the processing contents are determined in advance. When the camera unit 10 is 10B, 10D, or 10E, the control unit 551 appropriately refers to a table like FIG. 11 to determine the entity that performs processing on the image and the processing contents. In addition, when combining a plurality of camera units 10A, the control unit 551 can also refer to the above table in the same manner as when the camera unit is 10B (10E) or the camera unit 10D. Note that even when the camera unit 10 has one imaging unit 10 and when the camera unit 10 has four imaging units 10, the entity that performs processing on the image and the processing contents are determined in advance in a table, and the control unit 551 appropriately refers to the table.

[0082] In FIG. 11, the optical characteristics L-A of the photographing lens 210 are, for example, an angle of view of 200 degrees and an aperture F value of 2.0. Also, the optical characteristics L-B of the photographing lens 310 are, for example, an angle of view of 190 degrees and an aperture F value of 2.8. Furthermore, the specifications S-A of the imaging sensor 220 are, for example, a pixel count of 20M pixels and a size of 1 / 1.7 type. Also, the specifications S-B of the imaging sensor 320 are, for example, a pixel count of 40M pixels and a size of 1 type.

[0083] When there are differences in the optical characteristics of the imaging lenses 210 and 310 or differences in the specifications of the imaging sensors 220 and 320 between the imaging unit 10-1a and the imaging unit 10-1b, due to these differences, there will be differences in the number of data, signal values (brightness of the image, color tone of the image), degree of image blurring, etc. between the first image acquired by the imaging unit 10-1a and the second image acquired by the imaging unit 10-1b. Images with such differences are, for example, inappropriate for use in viewing two images as a stereo-formatted image, or when stitching the two images together by stitching processing, the boundary part may become discontinuous, etc. There may be cases where the usability in performing the second-stage signal processing is not good. Therefore, the first-stage signal processing (first processing α, α-1, α-2,...) mainly performed by the first processing unit 520 of the base unit includes processing for suppressing the differences between the two images, specifically, signal processing for making the color tone, image quality, resolution, etc. between the images closer.

[0084] Also, in FIG. 11, the positional relationship P(10-3) of the imaging units and the type B(10-3) of the base unit indicate the geometric information between the imaging unit 10-1a and the imaging unit 10-1b, for example, whether the orientations of the imaging unit 10-1a and the imaging unit 10-1b are the same or opposite. Depending on whether the orientations of the imaging unit 10-1a and the imaging unit 10-1b are the same or opposite, the use of whether the first image acquired by the imaging unit 10-1a and the second image acquired by the imaging unit 10-1b are viewed as a stereo-formatted image or as an omnidirectional image is different. Therefore, the second-stage signal processing (second processing β, β-1, β-2,...) mainly performed by the cloud 50 includes processing adapted to the uses of the two images.

[0085] Specifically, the second processing β includes stitching processing. Also, the second processing β-1 and the second processing β-2 include conversion processing to an orthographic cylindrical projection image and conversion processing to a stereo format. Note that the second processing β-1 and the second processing β-2 may include electronic gimbal processing, lens distortion correction processing, depth calculation processing, etc. as required.

[0086] FIG. 12 is a diagram for explaining the details of the second process β including the stitching process. In the above description, an example where the angle of view of the photographing lens 210 is 200 degrees and the angle of view of the photographing lens 310 is 190 degrees and the stitching process is possible has been illustrated. However, even if both the photographing lenses 210 and 310 are fish-eye lenses, the angle of view may not be 190 degrees as described above. In this case, if the sum of the angles of view of the photographing lens 210 and the photographing lens 310 is smaller than, for example, 370 degrees, the overlap when joining the two images is not sufficient, and it becomes difficult to perform the stitching process. Therefore, the control unit 551 switches the processing content as shown in FIG. 12 based on the angles of view of the photographing lens 210 and the photographing lens 310.

[0087] (Example 1) When the angle of view of the photographing lens 210 is 180 degrees or less than 180 degrees and the angle of view of the photographing lens 310 is 180 degrees or less than 180 degrees, the sum of the angles of view of the photographing lens 210 and the photographing lens 310 is smaller than 370 degrees. That is, it is difficult to obtain a VR360 omnidirectional (full sphere) image by joining the two images. Therefore, the control unit 551 determines that one of the processes from (i) to (iii) below will be performed by the server 50 as the second process β. (i) Set an error flag. (ii) After combining the two images, fill in the insufficient part of the omnidirectional image with black in a circular shape (referred to as a black band). (iii) Paste another image on the black band part of (ii) (referred to as wiping). Wiping is a process based on the idea that it looks better than the black band.

[0088] (Example 2) When the angle of view of the photographing lens 210 is from 180 degrees to 200 degrees, or 200 degrees or more, and the angle of view of the photographing lens 310 is less than 180 degrees, the sum of the angles of view of the photographing lens 210 and the photographing lens 310 is larger than 370 degrees. That is, it is possible to obtain a VR360 omnidirectional (full sphere) image by joining the two images. The control unit 551 determines that the stitching process will be performed by the server 50 as the second process β. On the other hand, when the sum of the field angles of the photographing lens 210 and the photographing lens 310 is not greater than 370 degrees, it is difficult to obtain a VR360 omnidirectional (full sphere) image by joining the two images. Therefore, the control unit 551 determines that one of the three processes from (i) to (iii) above is to be performed by the server 50 as the second process β.

[0089] (Example 3) When the field angle of the photographing lens 310 is from 180 degrees to 200 degrees, or 200 degrees or more, and the field angle of the photographing lens 210 is less than 180 degrees, the sum of the field angles of the photographing lens 210 and the photographing lens 310 is greater than 370 degrees. That is, a VR360 omnidirectional (full sphere) image can be obtained by joining the two images. The control unit 551 determines that the stitching process is to be performed by the server 50 as the second process β. On the other hand, when the sum of the field angles of the photographing lens 210 and the photographing lens 310 is not greater than 370 degrees, it is difficult to obtain a VR360 omnidirectional (full sphere) image by joining the two images. Therefore, the control unit 551 determines that one of the three processes from (i) to (iii) above is to be performed by the server 50 as the second process β. (Example 4) When the field angle of the photographing lens 210 is greater than 180 degrees and the field angle of the photographing lens 310 is greater than 180 degrees, a VR360 omnidirectional (full sphere) image can be obtained by joining the two images. The control unit 551 determines that the stitching process is to be performed by the server 50 as the second process β.

[0090] According to the first embodiment described above, the following operational effects can be obtained. (1) In the imaging system 1 having the camera unit 10 and the cloud 50, the camera unit 10 of the imaging system 1 includes an imaging unit 10-1 that acquires an image, a base unit 10-2 combined with the imaging unit 10-1, a first processing unit 520 included in the base unit 10-2 that performs signal processing on the image, a communication unit 550 that outputs information regarding the imaging unit 10-1 and the base unit 10-2 to the cloud 50, and a communication unit 550 that inputs the processing content determined by the cloud 50 from the cloud 50. The first processing unit 520 performs the processing of the processing content input to the communication unit 550. Further, the cloud 50 of the imaging system 1 includes a communication unit 556 that inputs information regarding the imaging unit 10-1 and the base unit 10-2 from the camera unit 10, a control unit 551 that determines the processing content for the image acquired by the imaging unit 10-1 based on the information input to the communication unit 556, and a communication unit 556 that outputs the processing content determined by the control unit 551 to the camera unit 10. With such a configuration, the camera unit 10 can notify the cloud 50 of the specifications of the imaging unit 10-1 and the state of the base unit 10-2 (for example, the state of the processing load in the first processing unit 520). On the other hand, the control unit 551 of the cloud 50 can flexibly determine the content of the signal processing for the image acquired by the imaging unit 10-1 based on the specifications of the imaging unit 10-1 and the state of the base unit 10-2, and can notify the camera unit 10.

[0091] (2) The processing content input by the communication unit 550 of the camera unit 10 includes the processing entity that performs the processing. With such a configuration, it is possible to know whether the input processing content is processing to be performed by the camera unit 10.

[0092] (3) The communication unit 550 of the camera unit 10 outputs information indicating the positional relationship between the imaging units 10-1a and 10-1b when a plurality of imaging units 10-1a and 10-1b are combined, as information regarding the imaging unit 10-1 and the base unit 10-2. With this configuration, it is useful information for determining the processing content for the first image acquired by the imaging unit 10-1a and the second image acquired by the imaging unit 10-1b. For example, it is possible to notify the cloud 50 whether the orientations of the imaging unit 10-1a and the imaging unit 10-1b are the same or opposite.

[0093] (4) The base unit 10-2 (or 10-3, 10-4) of the camera unit 10 houses the battery 570 as the power source of the camera unit 10, and the communication unit 550 outputs information indicating the remaining amount of the battery 570 as information regarding the imaging unit 10-1 and the base unit 10-2. With this configuration, it is possible to notify the cloud 50, which determines the processing content for the image, of the information useful for the determination.

[0094] (5) The imaging unit 10-1 of the camera unit 10 includes an imaging sensor 220 and a photographing lens 210 that guides light from the subject to the imaging sensor 220, and the communication unit 550 outputs information indicating the number of pixels of the imaging sensor 220 and the angle of view of the photographing lens 210 as information regarding the imaging unit 10-1 and the base unit 10-2. With this configuration, it is possible to notify the cloud 50, which determines the processing content for the image, of the information useful for the determination.

[0095] (6) In the imaging system 1 having the camera unit 10 and the cloud 50, the camera unit 10 of the imaging system 1 includes an imaging unit 10-1 that acquires an image, a base unit 10-2 combined with the imaging unit 10-1, a first processing unit 520 included in the base unit 10-2 that performs signal processing on the image, a communication unit 550 that outputs information regarding the imaging unit 10-1 and the base unit 10-2 to the cloud 50, and a communication unit 550 that inputs the processing content determined by the cloud 50 from the cloud 50. The first processing unit 520 performs the processing of the processing content input to the communication unit 550. Further, the cloud 50 includes a communication unit 556 that inputs information regarding the imaging unit 10-1 and the base unit 10-2 from the camera unit 10, a control unit 551 that determines the processing content for the image acquired by the imaging unit 10-1 based on the information input to the communication unit 556, and a communication unit 556 that outputs the processing content determined by the control unit 551 to the camera unit 10. With such a configuration, the camera unit 10 can notify the cloud 50 of the specifications of the imaging unit 10-1 and the state of the base unit 10-2 (for example, the state of the processing load in the first processing unit 520). On the other hand, the control unit 551 of the cloud 50 can flexibly determine the content of the signal processing for the image acquired by the imaging unit 10-1 based on the specifications of the imaging unit 10-1 and the state of the base unit 10-2.

[0096] (7) The processing content determined by the control unit 551 of the cloud 50 includes the processing entity that performs the processing. With such a configuration, it is possible to flexibly determine the content of the signal processing for the image acquired by the imaging unit 10-1, including where to perform the determined processing content.

[0097] (8) In the imaging system 1, the camera unit 10 and the cloud 50 are connected via a network line. With such a configuration, the camera unit 10 and the cloud 50 can be arranged at positions separated from each other.

[0098] (9) The cloud 50 of the imaging system 1 having the camera unit 10 and the cloud 50 includes an imaging unit 10-1 that acquires images, a base unit 10-2 combined with the imaging unit 10-1, and a first processing unit 520 that is included in the base unit 10-2 and performs signal processing on the images acquired by the imaging unit 10-1. It also includes a communication unit 556 that inputs information regarding the imaging unit 10-1 and the base unit 10-2 from the camera unit 10, a control unit 551 that determines the processing content to be performed by the first processing unit 520 of the camera unit 10 based on the information input to the communication unit 556, and a communication unit 556 that outputs the processing content determined by the control unit 551 to the camera unit 10. With such a configuration, the camera unit 10 can notify the cloud 50 of the specifications of the imaging unit 10-1 and the state of the base unit 10-2 (for example, the state of the processing load in the first processing unit 520). On the other hand, the control unit 551 of the cloud 50 can flexibly determine the content of the signal processing for the images acquired by the imaging unit 10-1 based on the specifications of the imaging unit 10-1 and the state of the base unit 10-2, and notify the camera unit 10.

[0099] (10) The processing content determined by the control unit 551 of the cloud 50 includes the processing entity that performs the processing. With such a configuration, it is possible to notify the camera unit whether the determined processing content is processing to be performed by the camera unit 10.

[0100] The following deformations are also within the scope of the present invention, and it is also possible to combine one or more of the deformation examples with the above-described embodiments. (Modification Example 1 of the First Embodiment) Regarding the multi-eye image processing when using, for example, a wide-angle lens (Wide) with a shooting angle of view of about 70 degrees or a telephoto lens (Tele) with a shooting angle of view of about 10 degrees as the shooting lens 210 of the first imaging unit 10-1a and the shooting lens 310 of the second imaging unit 10-1b, an explanation will be given.

[0101] FIG. 13 is a diagram for explaining a second process β-3 as an example of signal processing in the second stage when a photographing lens different from the fish-eye lens is used. In Modification 1, reference is made to the portion surrounded by the thick line in FIG. 13, that is, the photographing lens 210 of the first imaging unit 10-1a and the photographing lens 310 of the second imaging unit 10-1b, which are wide-angle (Wide) or telephoto (Tele) lenses.

[0102] In Modification 1, the control unit 551 switches the processing content as follows based on the angle of view of the photographing lens 210 and the photographing lens 310, and the geometric information and tilt information between the imaging unit 10-1a and the imaging unit 10-1b. It is assumed that the specifications of the imaging sensor 220 and the imaging sensor 320 are the same. (Example 1) When the photographing lens 210 is a wide-angle lens (Wide) and the photographing lens 310 is a wide-angle lens (Wide), and their optical characteristics are the same, or when the photographing lens 210 is a telephoto lens (Tele) and the photographing lens 310 is a telephoto lens (Tele), and their optical characteristics are the same, the control unit 551 determines that one of the processes from (i) to (iv) below will be performed by the server 50 as the second process β-3.

[0103] (i) High-pixel synthesis The images acquired by the imaging unit 10-1a and the imaging unit 10-1b are assumed to be of the same subject under the same shooting conditions. The server 50 (the second processing unit 553) shifts and arranges one of the images acquired by the first imaging unit 10-1a (referred to as the first image) and the image acquired by the second imaging unit 10-1b (referred to as the second image) by half of the pixel pitch (interval) of the imaging sensor 220 and the imaging sensor 320, and synthesizes the shifted first image and second image, thereby effectively halving the pixel pitch and increasing the resolution. (ii) Stereo image generation The images acquired by the imaging unit 10-1a and the imaging unit 10-1b are assumed to be of the same subject photographed under the same shooting conditions. The server 50 (the second processing unit 553) obtains a stereo-form image using the images acquired by the first imaging unit 10-1a and the second imaging unit 10-1b as the left-eye image and the right-eye image, respectively, based on the geometric information and tilt information between the first imaging unit 10-1a and the second imaging unit 10-1b.

[0104] (iii) HDR (High Dynamic Range) The images acquired by the imaging unit 10-1a and the imaging unit 10-1b are assumed to be of the same subject photographed with different exposures between the imaging unit 10-1a and the imaging unit 10-1b. The server 50 (the second processing unit 553) effectively expands the dynamic range of the image signal based on the image signals at the corresponding pixel positions of the image acquired by the first imaging unit 10-1a and the image acquired by the second imaging unit 10-1b. (iv) Depth calculation The images acquired by the imaging unit 10-1a and the imaging unit 10-1b are assumed to be of the same subject photographed under the same shooting conditions. The server 50 (the second processing unit 553) obtains the distance to the object using the principle of triangulation based on the image acquired by the imaging unit 10-1a and the image acquired by the imaging unit 10-1b.

[0105] (Example 2) When the photographing lens 210 is a wide-angle lens (Wide) and the photographing lens 310 is a telephoto lens (Tele), and their optical characteristics do not match, or when the photographing lens 210 is a telephoto lens (Tele) and the photographing lens 310 is a wide-angle lens (Wide), and their optical characteristics do not match, the control unit 551 determines that the server 50 performs the following process as the second process β-3.

[0106] In the case of the above (Example 2), it is assumed that one of the imaging units 10-1a and 10-1b having a telephoto lens (Tele) captures a subject included in the image captured by the other imaging unit having a wide-angle lens (Wide). The server 50 (the second processing unit 553) cuts out an image of a region corresponding to the image (referred to as a telephoto image) captured by the imaging unit having the telephoto lens (Tele) from the image (referred to as a wide-angle image) acquired by the other imaging unit having the wide-angle lens (Wide) to match the angle of view (denoted as "zoom" in FIG. 13), adjusts the resolution between the cut-out image and the telephoto image, and then obtains stereo-form images for the left eye and the right eye based on the geometric information and tilt information between the imaging unit 10-1a and the imaging unit 10-1b.

[0107] According to the modified example 1 described above, even when a wide-angle lens (Wide) or a telephoto lens (Tele) is used as the imaging lens 210 of the first imaging unit 10-1a and the imaging lens 310 of the second imaging unit 10-1b, the imaging system 1 can appropriately perform the second-stage signal processing on the multi-eye images captured by the plurality of imaging units 10-1.

[0108] (Modified Example 2 of the First Embodiment) A case where an infrared (IR: Infrared) sensor that receives infrared light is used as the imaging sensor 220 (320) of the first imaging unit 10-1a or the second imaging unit 10-1b will be described. In Modified Example 2, reference is made to the portion surrounded by the dashed line in FIG. 13, that is, the portion where the infrared sensor is mounted on one of the first imaging unit 10-1a and the second imaging unit 10-1b. Also, the angle of view of the imaging lens 210 (310) of the imaging unit equipped with the infrared sensor as the imaging sensor is assumed to be narrower than the angle of view of the telephoto lens (Tele).

[0109] In Modification 2, the control unit 551 switches the processing content as follows based on the angle of view of the imaging lens 210 and the imaging lens 310, and the geometric information and tilt information between the imaging unit 10-1a and the imaging unit 10-1b. (Example 1) When the imaging lens 210 of the first imaging unit 10-1a is a wide-angle lens (Wide), or when the imaging lens 310 of the second imaging unit 10-1b is a wide-angle lens (Wide) and the imaging sensor of the other imaging unit is an infrared sensor, the control unit 551 determines that the following processing is to be performed by the server 50 as the second processing β-3.

[0110] The server 50 (the second processing unit 553) cuts out the image of the area corresponding to the image (referred to as the thermal distribution image) acquired by the imaging unit having the infrared sensor from the image (referred to as the wide-angle image) acquired by the imaging unit having the wide-angle lens (Wide), adjusts the angle of view to match (denoted as "deformation" in FIG. 13), and after adjusting the resolution between the cut-out image and the thermal distribution image, based on the geometric information and tilt information between the imaging unit 10-1a and the imaging unit 10-1b, at least one of the images is deformed so that the positional relationship of the object in both images is matched. In this case, the center of the angle of view of the thermal distribution image coincides with the center of the cut-out image.

[0111] (Example 2) When the imaging lens 210 of the first imaging unit 10-1a is a telephoto lens (Tele), or when the imaging lens 310 of the second imaging unit 10-1b is a telephoto lens (Tele) and the imaging sensor of the other imaging unit is an infrared sensor, the control unit 551 determines that the following processing is to be performed by the server 50 as the second processing β-3.

[0112] The server 50 (the second processing unit 553) extracts an image of the overlapping area from the image acquired by the imaging unit having the telephoto lens (Tele) (referred to as the telephoto image) and the image acquired by the imaging unit having the infrared sensor (referred to as the thermal distribution image), aligns the angles of view of the extracted telephoto image and the extracted thermal distribution image (denoted as "center shift" in FIG. 13), and adjusts the resolutions of both. Then, based on the geometric information and tilt information between the imaging unit 10-1a and the imaging unit 10-1b, the image is deformed so as to align the positional relationship of the object in both images. In this case, since the center of the angle of view of the thermal distribution image and the center of the extracted image do not necessarily coincide, it is denoted as "center shift".

[0113] According to the modification 2 described above, even when an infrared sensor is used for the first imaging unit 10-1a or the second imaging unit 10-1b, the imaging system 1 can appropriately perform the second-stage signal processing on the multi-view images captured by the plurality of imaging units 10-1.

[0114] (Second Embodiment) <Overview of Imaging System> The camera unit 10 may be provided in the vehicle 3. In the second embodiment of the invention, an imaging system 1C that monitors the inside and outside of the vehicle 3 to control the running of the moving vehicle will be described. FIG. 14 is a schematic diagram showing an example of the imaging system 1C according to the second embodiment. The imaging system 1C includes six camera units 10A-11 to 10A-16 corresponding to the camera unit 10A in FIG. 1 and a cloud 50 (not shown). The camera units 10A-11 to 10A-13 are arranged inside the moving vehicle 3, and the camera units 10A-14 to 10A-16 are arranged outside the vehicle 3. The number of camera units 10A is not limited to the six shown and may be increased or decreased.

[0115] (1) Camera Unit Outside the Vehicle The vehicle 3 is assumed to have the right side of the drawing as the front and the left side of the drawing as the rear. The camera unit 10A-16 is composed of an imaging unit 10-1r and a base unit 10-12, and the imaging unit 10-1r is mounted on the base unit 10-12 provided at the front part of the vehicle 3. The imaging unit 10-1r using a fisheye lens with a shooting angle of view of 190 degrees shoots a range wider than a hemisphere centered on the forward direction of the vehicle 3.

[0116] The camera unit 10A-15 is composed of an imaging unit 10-1q and a base unit 10-13, and the imaging unit 10-1q is mounted on the base unit 10-13 provided at the rear part of the vehicle 3. The imaging unit 10-1q using a fisheye lens with a shooting angle of view of 190 degrees shoots a range wider than a hemisphere centered on the rearward direction of the vehicle 3.

[0117] The camera unit 10A-14 is composed of an imaging unit 10-1p and a base unit 10-11, and the imaging unit 10-1p is mounted on the base unit 10-11 provided on the roof part of the vehicle 3. The imaging unit 10-1p using a fisheye lens with a shooting angle of view of 190 degrees shoots a range wider than a hemisphere centered on the upward direction of the vehicle 3.

[0118] The vehicle 3 is an autonomous vehicle in which driving control including driving, stopping, and steering is performed by a driving control unit 4. The passengers of the vehicle 3 do not need to perform driving operations. When an image acquired by the camera units 10A-14 to 10A-16, that is, an image of the surroundings outside the vehicle 3, is transmitted from the vehicle 3 to the cloud 50, a driving control command calculated on the cloud 50 side based on the transmitted image is transmitted to the vehicle 3. The driving control unit 4 of the vehicle 3 controls the driving of the vehicle 3 based on the driving control command from the cloud 50.

[0119] (2) In-vehicle camera unit The camera unit 10A-12 is composed of the imaging unit 10-1n and the base unit 10-11. The base unit 10-11 extends downward along a center pillar (not shown) from the roof portion, and the imaging unit 10-1n is mounted at the extended position. The imaging unit 10-1n using a fisheye lens with a shooting angle of view of 190 degrees shoots the interior of the vehicle 3 wider than a hemispherical sky centered on the front direction perpendicular to the drawing plane.

[0120] The camera unit 10A-11 is composed of the imaging unit 10-1m and the base unit 10-11. The imaging unit 10-1m is mounted below the base unit 10-11 and near the ceiling of the rear seat. The imaging unit 10-1m shoots the passengers (especially the faces) sitting in the rear seat.

[0121] The camera unit 10A-13 is composed of the imaging unit 10-1o and the base unit 10-11. The imaging unit 10-1o is mounted below the base unit 10-11 and near the ceiling of the front seat. The imaging unit 10-1o shoots the passengers (especially the faces) sitting in the front seat.

[0122] When the images acquired by the camera units 10A-11 to 10A-13, that is, the images of the interior of the vehicle 3 are transmitted from the vehicle 3 to the cloud 50, the presence or absence of abnormalities of the passengers in the vehicle 3 is checked on the cloud 50 side based on the transmitted images. In addition, the images of the faces of the passengers acquired by the camera units 10A-11 and 10A-13 are used for person identification processing. Based on the images acquired by the in-vehicle camera units 10A-11 to 10A-13, a configuration for performing a three-dimensional recording of the in-vehicle space may be adopted. When having a meeting inside the vehicle, having a video conference with the outside, during vehicle-to-vehicle communication when moving among a plurality of vehicles 3, etc., and for security purposes, etc., three-dimensional video can be utilized.

[0123] <First Process and Second Process> In the second embodiment, when the control unit 551 of the server 50A (50B) of the cloud 50 determines the content of the first-stage signal processing (first processing) and the second-stage signal processing (second processing), the entity that executes the first processing (camera unit or cloud 50), and the entity that executes the second processing (camera unit or cloud 50), it uses the information about the camera units 10A-11 to 10A-16. (1) First processing The control unit 551 determines that, for the moving images respectively acquired by the camera units 10A-11 to 10A-16, the processes of compressing the moving images and converting them into images in the orthographic cylindrical projection method are to be performed as the first processing α-10 by the respective camera units 10A-11 to 10A-16. As a result, the first processing units 520 of the base units 10-11, base unit 10-13, and base unit 10-12, which are the base units of the camera units 10A-11 to 10A-16, perform the first processing α-10 on the moving images respectively acquired by the camera units 10A-11 to 10A-16.

[0124] (2) Second processing (2-1) Calculate the inter-vehicle distance The control unit 551 determines that the process of calculating the inter-vehicle distance with other vehicles based on the moving images respectively acquired by the camera unit 10A-16 provided in front of the vehicle 3 and the camera unit 10A-15 provided behind the vehicle 3 is to be performed as the second processing β-10 by the cloud 50. As a result, the second processing unit 553 of the server 50A (50B) of the cloud 50 calculates the inter-vehicle distance, for example, by counting the number of sets of white lines and blank intervals of the lane boundary lines on the highway. Specifically, when the lane boundary line is composed of a 20m total of an 8m white line and a 12m blank interval, and there are 5 sets of white lines and blank intervals captured between the vehicle and other vehicles, the inter-vehicle distance of 100m can be calculated. Note that the inter-vehicle distance with other vehicles may be calculated based on the moving images acquired by a stereo camera, or the inter-vehicle distance with other vehicles may be obtained using a millimeter-wave radar device or the like.

[0125] (2-2) Determine obstacles The control unit 551 determines to perform, as the second process β-11 in the cloud 50, the process of determining obstacles or the like located in the traveling direction of the vehicle 3 based on the moving images respectively acquired by the camera unit 10A-16 provided in front of the vehicle 3 or the camera unit 10A-15 provided behind the vehicle 3. Thereby, the second processing unit 553 of the server 50A (50B) of the cloud 50 detects obstacles or the like on the road and detects depressions or the like on the road. The control unit 551 can issue a driving control command so as to avoid the detected obstacles and abnormalities of the road.

[0126] (2-3) Generate operation information The control unit 551 determines to perform, as the second process β-12 in the cloud 50, the process of detecting an operation on the operation member of the vehicle 3 based on the moving image acquired by the camera unit 10A-12 provided inside the vehicle 3 and generating corresponding operation information. Thereby, the second processing unit 553 of the server 50A (50B) of the cloud 50 detects an operation by the occupant of the vehicle 3 on an operation member or the like. The control unit 551 can perform a process pre-associated with the detected operation, such as issuing a driving control command for an emergency stop.

[0127] (2-4) Generate a panoramic image The control unit 551 determines to perform, as the second process β-13 in the cloud 50, the process of generating a panoramic image representing the scenery around the vehicle 3 in a 360-degree panoramic form based on the moving image acquired by the camera unit 10A-14 provided on the roof of the vehicle 3, for example. Thereby, the second processing unit 553 of the server 50A (50B) of the cloud 50 generates a panoramic image suitable for driving records and the like.

[0128] (2-5) Identify the occupant The control unit 551 determines to perform, as the second process β-14 in the cloud 50, the process of identifying the occupant of the vehicle 3 based on the moving images acquired by the camera units 10A-11 and 10A-13, for example. As a result, the face detection unit 554 and the face recognition unit 555 of the server 50A (50B) of the cloud 50 detect faces in the moving image, and identify the persons of the detected faces based on the names and faces registered in advance in the storage unit 558. When the occupant in the vehicle is not a person registered in advance, the control unit 551 can take measures such as not permitting the vehicle 3 to travel with respect to the travel control unit 4 of the vehicle 3, for example.

[0129] (6) Determine an event The control unit 551 determines to perform, as the second process β-15 in the cloud 50, an event determination process including detection of an occupant's sudden illness based on the moving image acquired by the camera units 10A-11 and 10A-13 and detection of a trouble in the vehicle based on the moving image acquired by the camera unit 10A-12. As a result, the second processing unit 553 of the server 50A (50B) of the cloud 50 detects the occurrence of a sudden illness in the vehicle 3 or the occurrence of a trouble in the vehicle 3. The control unit 551, for example, confirms the safety of the occupant with the vehicle 3 via the communication unit 556, or sends a command to the travel control unit 4 to evacuate the vehicle 3 to a safe place.

[0130] In the description of the above-described second embodiment, the case where most of the second process is performed by the server 50 is illustrated. However, when the vehicle 3 is traveling in a tunnel or in an underground facility, or depending on the communication situation, there may be a situation where real-time communication cannot be performed between the vehicle 3 and the cloud 50. When communication between the vehicle 3 and the cloud 50 cannot be established, the second processing unit 530 of the base units 10-11, 10-13, and 10-12, which are the base units of the camera units 10A-11 to 10A-16, shall perform the second process that the server 50 was supposed to perform, instead of the server 50. Further, a management unit for managing the second processing unit 530 of the base units 10-11, 10-13, and 10-12 may be provided in advance, and the management unit may manage the second process in the vehicle 3 when communication between the vehicle 3 and the cloud 50 cannot be established.

[0131] Note that the second processes respectively performed by the camera units 10A-11 to 10A-16 and the server 50 may be changed as appropriate. Then, based on the state of the processing load in the camera units 10A-11 to 10A-16, the main bodies of the first process α-10 and the second processes β-10 to β-14 are changed. Further, when real-time communication cannot be established between the vehicle 3 and the cloud 50, the camera units 10A-11 to 10A-16 will be the main bodies to perform the first process α-10 and the second processes β-10 to β-14.

[0132] According to the above-described second embodiment, the following operational effects can be obtained. (1) The camera units 10A-16 etc. of the imaging system 1C having the camera units 10A-16 etc. and the cloud 50 include an imaging unit 10-1r etc. for acquiring an image, a base unit 10-12 etc. combined with the imaging unit 10-1r etc., a first processing unit 520 included in the base unit 10-12 etc. for performing signal processing on the image, a communication unit 550 for outputting information regarding the imaging unit 10-1r etc. and the base unit 10-12 etc. to the cloud 50, and a communication unit 550 for inputting the processing content determined by the cloud 50 from the cloud 50. The first processing unit 520 performs the processing of the processing content input to the communication unit 550. Further, the cloud 50 includes a communication unit 556 for inputting information regarding the imaging unit 10-1r etc. and the base unit 10-12 etc. from the camera units 10A-16 etc., a control unit 551 for determining the processing content for the image acquired by the imaging unit 10-1r etc. based on the information input to the communication unit 556, and a communication unit 556 for outputting the processing content determined by the control unit 551 to the camera units 10A-16 etc. With such a configuration, the camera units 10A-16 etc. can notify the cloud 50 of the specifications of the imaging unit 10-1r etc. and the state of the base unit 10-12 etc. (for example, the state of the processing load in the first processing unit 520). On the other hand, the control unit 551 of the cloud 50 can flexibly determine the content of the signal processing for the image acquired by the imaging unit 10-1r etc. based on the specifications of the imaging unit 10A-16 etc. and the state of the base unit 10-12 etc.

[0133] (2) The processing content determined by the control unit 551 of the cloud 50 includes the processing entity that performs the processing. With such a configuration, it is possible to flexibly determine the content of signal processing for the images acquired by the imaging unit 10-1r, etc., including where the determined processing content is to be performed.

[0134] (3) The imaging system 1C is configured such that the camera units 10A-16, etc. and the cloud 50 are connected via a network line. With such a configuration, even when the camera units 10A-16, etc. (vehicle 3) move to a position away from the cloud 50, information can be appropriately input and output.

[0135] (Third Embodiment) <Overview of the Imaging System> The camera unit 10 may be provided inside the office. In the third embodiment of the invention, an imaging system 1D for monitoring to prevent unregistered persons from entering the office will be described. FIG. 15 is a schematic diagram showing an example of the imaging system 1D according to the third embodiment. The imaging system 1D includes four camera units 10A-21 to 10A-24 corresponding to the camera unit 10A in FIG. 1 and a server 50A of the cloud 50. The number of camera units 10 is not limited to the four shown and may be increased or decreased, and the number of servers 50A constituting the cloud 50 is not limited to the one shown. Also, although an embodiment where each camera unit 10 and the server 50A are directly connected is shown, as in the case of FIG. 1, each camera unit 10 and the server 50A may be connected via a wired or wireless network communication network.

[0136] <First Process and Second Process> In the third embodiment, when the control unit 551 of the server 50A of the cloud 50 determines the content of the first process and the second process, the entity that executes the first process (camera unit 10 or cloud 50), and the entity that executes the second process (camera unit 10 or cloud 50), it uses information regarding the camera units 10A-21 to 10A-24.

[0137] In this embodiment, based on the frames of the moving images acquired by the camera units 10A-21 to 10A-24, the process of detecting the face shown is defined as the first process A. Further, the process of identifying the person of the face detected in the first process A based on the name and face registered in advance in the storage unit 558 is defined as the second process B.

[0138] The control unit 551 determines the main bodies of the first process A and the second process B as shown in the following (Example 1) to (Example 4) based on the state of the processing load in the camera units 10A-21 to 10A-24. The state of the processing load is determined by, for example, the occupancy rate of the process in the CPU or the like × the processing time and the frame rate (unit: fps) of the moving image to be acquired. The larger these numerical values are, the higher the state of the processing load, and the smaller these numerical values are, the lower the state of the processing load. Also, even if the content of the signal processing is the same, when the processing ability of the CPU or the like is high, the state of the processing load is low, and when the processing ability of the CPU or the like is low, the state of the processing load is high. The processing ability and the state of the processing load of the CPU or the like in each camera unit are included in the information regarding the corresponding camera units 10A-21 to 10A-24.

[0139] (Example 1) Perform the first process and the second process in the camera unit When information similar to the name and face information registered in advance in the storage unit 558 is stored in the second processing unit 530 and the processing load in the base unit 10-21 of the camera unit 10A-21 is equal to or lower than the first threshold value, the control unit 551 determines to perform the first process A in the first processing unit 520 of the base unit 10-21 and the second process B in the second processing unit 530 of the base unit 10-21. As a result, the first processing unit 520 detects a face based on the moving image (referred to as the original image) acquired by the imaging unit 10-1c of the camera unit 10A-21. The data of the detected face image (face image) is sent from the first processing unit 520 to the second processing unit 530. The second processing unit 530 identifies the person of the face in the face image based on the name and face registered in advance. In the case of Example 1, the data indicating the name (person's name) of the identified person is transmitted from the camera unit 10A-21 to the cloud 50.

[0140] (Example 2) Perform the first process in the camera unit and the second process in the cloud When the processing load in the base unit 10-22 of the camera unit 10A-22 is equal to or higher than the first threshold and lower than or equal to the second threshold, the control unit 551 determines to perform the first process A in the first processing unit 520 of the base unit 10-22 and the second process B in the server 50A (face recognition unit 555) of the cloud 50. As a result, the first processing unit 520 detects a face based on the moving image (original image) acquired by the imaging unit 10-1d of the camera unit 10A-22. The data of the detected face image (face image) is sent from the camera unit 10A-22 to the cloud 50. The face recognition unit 555 of the server 50A identifies the person of the face in the face image based on the name and face registered in advance in the storage unit 558.

[0141] (Example 3) Perform the first process in the camera unit and the second process in the cloud Although the processing load of the base unit 10-23 of the camera unit 10A-23 is equal to or lower than the first threshold, when information similar to the name and face information registered in advance in the storage unit 558 is not stored in the second processing unit 530, the control unit 551 determines to perform the first process A in the first processing unit 520 of the base unit 10-23 and the second process B in the server 50A (face recognition unit 555) of the cloud 50. The operation in this case is the same as that in Example 2. Therefore, the description of the operation for Example 3 is omitted.

[0142] (Example 4) Perform the first process and the second process in the cloud When the processing load of the base unit 10-24 of the camera unit 10A-24 is higher than the second threshold, the control unit 551 determines to perform the first process A by the server 50A (face detection unit 554) of the cloud 50 and the second process B by the server 50A (face recognition unit 555) of the cloud 50. Accordingly, the data of the moving image (= original image) acquired by the imaging unit 10-1f of the camera unit 10A-24 is transmitted from the camera unit 10A-24 to the server 50A of the cloud 50. The face detection unit 554 of the server 50A detects a face based on the original image. The face recognition unit 555 of the server 50A identifies the person of the face in the face image based on the name and face registered in advance in the storage unit 558. Note that the control unit 551 also determines to perform the first process A by the server 50A (face detection unit 554) of the cloud 50 and the second process B by the server 50A (face recognition unit 555) of the cloud 50 when a failure occurs in the processing unit of the base unit 10-24 of the camera unit 10A-24 and when no processing unit is provided in the base unit 10-24 of the camera unit 10A-24. Error information indicating the occurrence of a failure and information indicating the specifications of the device such as the absence of a processing unit are included in the information regarding the camera unit 10A-24.

[0143] According to the above-described third embodiment, the following operational effects can be obtained. (1) The camera unit 10A-21 etc. of the imaging system 1D having the camera unit 10A-21 etc. and the server 50A includes an imaging unit 10-1c etc. that acquires an image, a base unit 10-21 etc. combined with the imaging unit 10-1c etc., a first processing unit 520 included in the base unit 10-21 etc. that performs the first process A on the image, a second processing unit 530 that performs the second process B on the image, a communication unit 550 that outputs information regarding the imaging unit 10-1c etc. and the base unit 10-21 etc. to the server 50A, and a communication unit 550 that inputs the processing contents of the first process A and the second process B determined by the server 50A from the server 50A. The first processing unit 520 performs the first process A of the processing contents input to the communication unit 550. The second processing unit 530 performs the second process B of the processing contents input to the communication unit 550. Further, the server 50A includes a communication unit 556 that inputs information regarding the imaging unit 10-1c etc. and the base unit 10-21 etc. from the camera unit 10A-21 etc., a control unit 551 that determines the processing details of the first process A and the processing details of the second process B for the image acquired by the imaging unit 10-1c etc. based on the information input to the communication unit 556, and a communication unit 556 that outputs the processing details of the first process A and the processing details of the second process B determined by the control unit 551 to the camera unit 10A-21 etc. With this configuration, the camera unit 10A-21 etc. can notify the server 50A of the specifications of the imaging unit 10-1c etc. and the state of the base unit 10-21 etc. (for example, the state of the processing load in the first processing unit 520 and the second processing unit 530). On the other hand, the control unit 551 of the server 50A can flexibly determine the processing details of the first process A and the processing details of the second process B for the image acquired by the imaging unit 10-1c etc. based on the specifications of the imaging unit 10-1c etc. and the state of the base unit 10-21 etc.

[0144] (2) The processing details determined by the control unit 551 of the server 50A include the processing entity that performs the processing. With this configuration, it is possible to flexibly determine the processing details of the first process A and the processing details of the second process B for the image acquired by the imaging unit 10-1c etc., including where the determined processing details will be performed, either in the camera unit 10A-21 etc. or in the server 50A.

[0145] (3) The imaging system 1D is configured such that the camera unit 10A-21 etc. and the server 50A are connected via a network line. With this configuration, the camera unit 10A-21 etc. and the server 50A can be arranged at positions separated from each other.

[0146] (Modifications of the First to Third Embodiments) A configuration obtained by removing the camera unit 10 from the imaging system 1 etc. described above may be made to function as an image processing system. (1) The image processing system includes a second processing unit 553 that performs second-stage signal processing using an image acquired by an imaging unit 10-1 or the like and first-stage signal-processed by a first processing unit 520, and a control unit 551 that determines the processing content of the second processing unit 553 based on first information regarding the imaging unit 10-1 or the like and second information regarding the first processing unit 520. With such a configuration, for example, based on the specifications of the imaging unit 10-1 and the state of the first processing unit 520 or the like, the control unit 551 can appropriately determine the processing content of the second processing unit 553.

[0147] (2) The first processing unit 520 is provided in a base unit 10-2 or the like combined with the imaging unit 10-1 or the like. With such a configuration, based on the state of the base unit 10-2 or the like, the control unit 551 can appropriately determine the processing content of the second processing unit 553.

[0148] (3) The control unit 551 is connected to the base unit 10-2 or the like via a network line. With such a configuration, the control unit 551 and the base unit 10-2 or the like can be arranged at separated positions.

[0149] (4) The second processing unit 553 is connected to the base unit 10-2 or the like via a network line. With such a configuration, the second processing unit 553 and the base unit 10-2 or the like can be arranged at separated positions.

[0150] <Program> Record a program that causes the control unit 551 of the server 50A of the above image processing system, that is, the process illustrated in FIG. 10, to be executed on a computer-readable recording medium, and supply the program recorded on this recording medium to the server 50A (50B). The "computer-readable recording medium" refers to portable recording media such as flexible disks, magneto-optical disks, optical disks, memory cards, etc., and storage devices such as hard disks built into computer systems. The computer system shall include an OS (Operating System) and the hardware of peripheral devices, such as the illustrated server 50A (50B).

[0151] Also, the "computer-readable recording medium" may include those that hold a program dynamically for a short time, such as a communication line when transmitting a program via a network communication network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as the volatile memory inside a computer system that serves as a server or a client in that case. Further, the above program may be for realizing a part of the aforementioned functions, or may be for realizing the aforementioned functions in combination with a program already recorded in the computer system.

[0152] FIG. 16 is a diagram for explaining the supply of a program to the server 50A (50B). The server 50A (50B) can receive the provision of a program recorded on, for example, a CD-ROM 953. Also, the server 50A (50B) can receive the provision of a program via the communication line 900. The computer 952 is a server computer that provides the above program and stores the program in a recording medium 954 such as a hard disk. The communication line 900 is a communication line such as the Internet, personal computer communication, or a dedicated communication line. The computer 952 reads the program from the recording medium 954 and transmits the program to the server 50A (50B) via the communication line 900. That is, the program is carried by a carrier wave as a data signal and transmitted via the communication line 900. The server 50A (50B) receives the transmitted program. In this way, the program can be supplied as a computer-readable computer program product in various forms such as a recording medium and a carrier wave.

[0153] In the above, various embodiments and modifications have been described, but the present invention is not limited to these contents. Modes of using each configuration shown in the embodiments and modifications in combination are also included within the scope of the present invention. Other modes conceivable within the scope of the technical idea of the present invention are also included within the scope of the present invention.

Explanation of Reference Numerals

[0154] 1, 1A, 1B, 1C, 1D… Imaging system, 3… Vehicle, 4… Travel control unit, 10, 10A, 10B, 10C, 10D, 10E, 10A-1 to 10A-9, 10A-11 to 10A-16, 10A-21 to 10A-24, B-1 to B-6… Camera unit, 10-1, 10-1a, 10-1b, 10-1c to 10-1f, 10-1m to 10-1r… Imaging unit, 10-2 to 10-4, 10-11 to 10-13, 10-21 to 10-24… Base unit, 40… Station server, 50… Cloud, 50A, 50B, C-1, S-1 to S-3… Server, 100… Head-mounted display, 200… Terminal, 210, 310… Shooting lens, 220, 320… Imaging sensor, 510, 551… Control unit, 520, 552… First processing unit, 530, 553… Second processing unit, 540… Position detection unit, 550, 556… Communication unit, 554… Face detection unit, 555… Face recognition unit, 557, 560… Power supply unit, 558… Memory unit, 570… Battery, α, α-1, α-2, α-10, A… First processing, β, β-1, β-2, β-3, β-10 to β-15, B… Second processing

Claims

【Claim 1】 A first image acquisition unit that acquires an image; A second image acquisition unit that acquires an image; A communication unit that transmits first information regarding the relative positional relationship between the first image acquisition unit and the second image acquisition unit to an external device; A processing unit that performs processing on first data output from the first image acquisition unit, the processing being determined by the external device based on the first information An imaging system comprising the above components.

Images