Imaging system, data processing method and program

Abstract

To flexibly perform processing on an image.SOLUTION: A camera unit comprises: an image acquisition part which acquires an image; a base part which is combined with the image acquisition part; a processing part which is included in at least one of the image acquisition part and the base part and performs prescribed processing on the image; an output part which outputs information about the image acquisition part and the base part to an external device; and an input part which inputs the processing content determined on the basis of the information from the external device. The processing part performs processing of the processing content input to the input part.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to Photography Image system, Data processing methods and regarding the program. [Background technology]

[0002] There is known an image processing device that generates a composite image by synthesizing images captured by a surveillance camera (see Patent Document 1). In the conventional technology, it is difficult for the camera and the image processing device to flexibly process the images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-4630 A Summary of the Invention

[0004] According to the first aspect of the present invention Imaging System Get the image No. 1 An image acquisition unit; A second image acquisition unit that acquires an image; The above a communication unit that transmits first information regarding a relative positional relationship between a first image acquisition unit and the second image acquisition unit to an external device, and a processing unit that performs processing determined by the external device based on the first information on first data output from the first image acquisition unit. . According to the second aspect of the present invention The data processing method includes transmitting first information regarding a relative positional relationship between a first image acquisition unit that acquires an image of a subject and a second image acquisition unit that acquires an image of the subject to an external device, and the external device performs processing determined based on the first information on first data output from the first image acquisition unit. . According to the third aspect of the present invention The program causes the data processing device to function as a processing unit that transmits first information regarding the relative positional relationship between a first image acquisition unit that acquires an image of a subject and a second image acquisition unit that acquires an image of the subject to an external device, and performs processing determined by the external device based on the first information on first data output from the first image acquisition unit. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a schematic diagram illustrating an imaging system according to a first embodiment. [Diagram 2] FIG. 1 is a schematic diagram illustrating an imaging system provided with a station server. [Diagram 3] FIG. 3(a) is a perspective view illustrating the appearance of camera unit 10A, FIG. 3(b) is a perspective view illustrating the appearance of camera unit 10B, and FIG. 3(c) is a perspective view illustrating the appearance of camera unit 10D. [Figure 4] 2 is a block diagram illustrating the configuration of a camera unit 10A. FIG. [Diagram 5] FIG. 2 is a block diagram illustrating the configuration of a camera unit 10D. [Figure 6] FIG. 1 is a diagram showing an example of an arrangement of nine camera units. [Figure 7] FIG. 11 is a block diagram showing an example of a combination of a plurality of camera units. [Figure 8] FIG. 2 is a block diagram illustrating a configuration of a cloud server. [Figure 9] 10 is a flowchart illustrating a flow of processing executed by a control unit of the camera unit. [Figure 10] 11 is a flowchart illustrating a flow of processing executed by a control unit of the cloud. [Figure 11] FIG. 2 is a diagram showing entities that perform image processing and the contents of the processing. [Figure 12] FIG. 13 is a diagram illustrating processing for an image. [Figure 13] FIG. 13 is a diagram illustrating processing for an image. [Figure 14] FIG. 11 is a schematic diagram illustrating an imaging system according to a second embodiment. [Figure 15] FIG. 13 is a schematic diagram illustrating an imaging system according to a third embodiment. [Figure 16] FIG. 13 is a diagram illustrating program provision. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Hereinafter, an embodiment of the invention will be described with reference to the drawings. (First embodiment) <Overview of imaging system> 1 is a schematic diagram illustrating an imaging system 1 according to a first embodiment of the invention. The imaging system 1 is composed of multiple camera units 10 and multiple servers 50A and 50B called a cloud 50. As the camera units 10, camera units 10A, 10B, 10C, 10D, and 10E are illustrated. Each camera unit 10 is connected to the cloud 50 via a wired or wireless network communication network. Fig. 1 illustrates the case of wireless connection. The number of camera units 10 is not limited to five as shown in the figure and may be increased or decreased, and the number of servers constituting the cloud 50 is also not limited to two as shown in the figure and may be increased or decreased.

[0007] Note that the connection between the camera unit 10 and the cloud 50 via a communication network is just one example, and may be configured as a connection in a local network that is not connected to the Internet, or as a connection within a device that is not connected to a local network. When configured as a connection within a device, the camera unit 10 is connected to an image processing unit (which may be called an image processing device) instead of the cloud 50.

[0008] Each camera unit 10 has at least one photographing lens and an image sensor, and can capture one or multiple images for each camera unit 10. In the example of Fig. 1, camera units 10 are distinguished and given different reference symbols (10A, 10B, 10C, 10D, and 10E) when they differ in at least one of the number of photographing lenses that each camera unit 10 has, the optical characteristics of the optical members including the photographing lenses, and the presence or absence of a sound collecting microphone in the camera unit 10.

[0009] For example, camera unit 10A has one imaging unit and captures one image. Camera units 10B, 10D, and 10E have a first imaging unit and a second imaging unit and capture two images. Camera unit 10E has built-in sound collection microphones M1 and M2 for capturing sound, whereas camera unit 10B does not have a built-in sound collection microphone. Moreover, camera unit 10C has a first imaging section, a second imaging section, a third imaging section, and a fourth imaging section, and captures four images.

[0010] The image data captured by the camera units 10A, 10B, 10C, 10D, and 10E can be transmitted to the cloud 50 and stored in the server 50A or 50B.

[0011] The image captured by the camera unit 10 may be a still image or a video. The image may be a planar image called 2D, or a stereoscopic image called 3D (a sense of depth based on parallax). The 2D image includes an omnidirectional image called VR360. The 3D image includes a stereo image using parallax such as VR180. The VR180 image may be an image that complies with the format of live-action VR (Virtual Reality) with a 180-degree field of view up, down, left, and right proposed by Google (trademark), or may not be an image that complies with the format. The VR180 image can be acquired by a camera unit with two fisheye lenses arranged side by side as the shooting lens, as shown in the camera unit 10D, for example.

[0012] The user 2 can view 2D images stored in the cloud 50, for example, using a terminal 200 such as a smartphone. The user 2, who is registered in the cloud 50 in advance, connects the terminal 200 to the cloud 50 via a communication network and views any image stored in the cloud 50. The cloud 50 plays back data of the specified image and distributes it to the terminal 200 of the user 2. Note that storing images in the cloud 50 is just an example, and images may be stored in a camera unit other than the cloud 50, a station server described below, or terminal 200.

[0013] The user 2 can also view, for example, 3D images stored in the cloud 50 using the head mounted display 100. When the user 2 connects to the cloud 50 via a communication network, the cloud 50 reproduces data of a specified image and distributes it to the head mounted display 100 of the user 2.

[0014] The images stored in the server 50A or 50B of the cloud 50 may be used for monitoring purposes. For example, a monitoring device connected to the cloud 50 monitors for the presence or absence of abnormalities based on the images captured by each camera unit 10 and transmitted to the cloud 50.

[0015] An intermediate server 40 (which may be referred to as a station server) may be provided between the camera unit 10 and the server of the cloud 50. Fig. 2 is a schematic diagram illustrating an imaging system 1A in which a station server 40 is provided. The imaging system 1A is made up of six camera units B (B-1 to B-6) corresponding to the camera unit 10D in Fig. 1, a station server 40, and a cloud 50. The station server 40 is made up of, for example, three servers S-1 to S-3. The cloud 50 is made up of, for example, one server C-1. The number of camera units B is not limited to six as shown in the figure and may be increased or decreased, and the number of servers constituting station server 40 is not limited to three as shown in the figure and may be increased or decreased. The same applies to the number of servers constituting cloud 50.

[0016] The camera units B-1 to B-3 are respectively connected to the servers S-1 and S-2 of the station server 40. The servers S-1 and S-2 are mutually connected within the station server 40. Moreover, the camera units B-4 to B-6 are each connected to the server S-3 of the station server 40.

[0017] The dashed arrows between server S-2 and server S-3, between camera unit B-4 and server S-2, between camera unit B-3 and server S-3, and between camera unit B-3 and camera unit B-4 indicate paths that can be connected in an emergency, such as when server S-1 breaks down. When server S-1 is unavailable, the imaging system 1A can be operated in the same way as in normal times by connecting the emergency connection paths.

[0018] In the imaging system 1 of Fig. 1, the first stage signal processing for the images acquired by the camera units 10A, 10B, 10C, 10D, and 10E, and the second stage signal processing for the images after the first stage signal processing are mainly performed by the camera unit 10 that acquired the images, or mainly by the cloud 50 (servers 50A and 50B). In other words, the first and second stages of signal processing can be appropriately shared between the camera units 10A, 10B, 10C, 10D, and 10E and the cloud 50 (servers 50A and 50B). The sharing includes a case where the camera unit 10 is responsible for all of the first and second stages of signal processing, and a case where the signal processing is shared between the camera unit 10 and the cloud 50. 2, the imaging system 1A is configured so that the first stage signal processing for images acquired by camera units B-1, B-2, B-3, B-4, B-5, and B-6, and the second stage signal processing for images after the first stage signal processing, can be appropriately shared among camera unit Bn (n is 1 to 6) that acquired the images, station server 40 (servers S-1 to S-3), and cloud 50 (server C-1). Sharing includes cases where camera unit Bn and station server 40 share the first and second stages of signal processing, and cases where camera unit Bn, station server 40, and cloud 50 share the first and second stages of signal processing. The configurations of the servers S-1 to S-3 of the station server 40 are similar to the configuration of a server 50A (50B) of the cloud 50, which will be described in detail later. The configuration of the server C-1 is also similar to the configuration of the server 50A (50B). In the following description, a case where the first stage signal processing and the second stage signal processing are performed as the signal processing for an image will be exemplified, but the signal processing may be further divided into a plurality of stages, and each signal processing may be appropriately shared and performed between the camera unit 10 and the cloud 50. For example, the signal processing divided into the first stage to the fourth stage is shared and performed by the camera unit 10 and the cloud 50, etc. The imaging system 1 (imaging system 1A) will now be described in further detail.

[0019] <Camera unit overview> Fig. 3 is a schematic diagram illustrating an overview of camera unit 10. Fig. 3(a) is a perspective view illustrating the appearance of camera unit 10A, Fig. 3(b) is a perspective view illustrating the appearance of camera unit 10B, and Fig. 3(c) is a perspective view illustrating the appearance of camera unit 10D.

[0020] (1) Overview of Camera Unit 10A The camera unit 10A is an imaging device in which one imaging unit 10-1 is attached to a base unit 10-2. The base unit 10-2 can be fitted with one imaging unit 10-1. The imaging unit 10-1 includes a photographing lens, an imaging sensor, an aperture, a shutter mechanism, an optical zoom mechanism, a focus adjustment mechanism, a shake correction mechanism, an optical filter, etc. In this embodiment, a fisheye lens with a shooting angle of 190 degrees is used as the shooting lens of camera unit 10A. When a fisheye lens is used for imaging section 10-1 of camera unit 10A, it is possible to capture a wider range (190 degrees up, down, left, right) than a hemispherical image having a field of view of 180 degrees up, down, left, and right. Instead of a fisheye lens, an imaging unit equipped with a lens for a different purpose may be attached. For example, an imaging unit equipped with a standard lens with a shooting angle of about 50 degrees, a wide-angle lens with a shooting angle of about 70 degrees, or a telephoto lens with a shooting angle of about 10 degrees may be attached. A shooting lens with a narrow shooting angle is used for purposes that do not capture a hemispherical image.

[0021] (2) Overview of Camera Unit 10B The camera unit 10B is an imaging device in which two imaging units 10-1a and 10-1b are attached to a base unit 10-3. The base unit 10-3 can be fitted with the two imaging units 10-1a and 10-1b. In this embodiment, a fisheye lens with a shooting angle of 190 degrees is used as the imaging lens of the camera unit 10B. If a fisheye lens is used for the imaging units 10-1a and 10-1b of the camera unit 10B, it becomes possible to acquire the above-mentioned VR360 omnidirectional (so-called omnidirectional) image by stitching together two semi-spherical images. Instead of a fisheye lens, an imaging unit equipped with a different lens for different purposes may be attached. For example, an imaging unit equipped with a standard lens with a shooting angle of about 50 degrees, a wide-angle lens with a shooting angle of about 70 degrees, or a telephoto lens with a shooting angle of about 10 degrees can be attached. A shooting lens with a narrow shooting angle is used for purposes that do not capture VR360 images.

[0022] (3) Overview of Camera Unit 10D The camera unit 10D is an imaging device in which two imaging units 10-1a and 10-1b are attached to a base unit 10-4. The base unit 10-4 can be fitted with the two imaging units 10-1a and 10-1b. In this embodiment, a fisheye lens with a shooting angle of 190 degrees is used as the shooting lens of the camera unit 10D. When a fisheye lens is used for the imaging units 10-1a and 10-1b of the camera unit 10D, it becomes possible to obtain the above-mentioned VR180 image based on two hemispherical images. Instead of a fisheye lens, an imaging unit equipped with a lens for a different purpose may be attached. For example, an imaging unit equipped with a standard lens with a shooting angle of about 50 degrees, a wide-angle lens with a shooting angle of about 70 degrees, or a telephoto lens with a shooting angle of about 10 degrees may be attached. A shooting lens with a narrow shooting angle is used for purposes that do not capture a hemispherical image.

[0023] The camera unit 10B and the camera unit 10D have in common the fact that they both have two imaging units 10-1a and 10-1b. However, they differ in that the orientations of the two imaging units 10-1a and 10-1b mounted between the base unit 10-3 and the base unit 10-4 are different. That is, in the camera unit 10B, the two imaging units 10-1a and 10-1b photograph different subjects located in opposite directions, whereas in the camera unit 10D, the two imaging units 10-1a and 10-1b photograph a common subject located in the same direction.

[0024] (4) Overview of Camera Units 10C and 10E Although the perspective views of the camera unit 10C and the camera unit 10E are not shown, each has an appearance as exemplified in FIG. 1. The camera units 10A, 10B, 10C, 10D, and 10E are installed in various locations, outdoors and indoors, for the purpose of, for example, monitoring, observation, and appreciation. Outdoors, they are installed on supports or on the outer walls of buildings. Indoors, they are installed on ceilings, walls, and pillars. The camera unit 10 may also be mounted on a flying drone or the like, or may be installed in the cabin of an aircraft, ship, train, vehicle, or the like.

[0025] There are two installation methods: adding the entire camera unit 10, and embedding the base of the camera unit 10 in the ceiling, wall, etc. beforehand and then attaching the imaging unit 10-1 to the base as needed. It is also possible to remove the imaging unit 10-1 of the camera unit 10 that is already installed and replace it with another imaging unit 10-1 with the same specifications or another imaging unit 10-1 with different specifications. The imaging units 10 (10-1a, 10-1b, . . . ) have a common mount for mounting to a base unit, and can be easily mounted to any of the base units.

[0026] <Camera unit configuration> (1) Configuration of camera unit 10A The configuration of the camera unit 10A will be described with reference to the block diagram illustrated in Fig. 4. The camera unit 10A is composed of an imaging unit 10-1 and a base unit 10-2. The imaging unit 10-1 includes a photographing lens 210, an imaging sensor 220, and the above-mentioned aperture, shutter mechanism, etc. (not shown). The base unit 10-2 includes a control unit 510, a first processing unit 520, a second processing unit 530, a position detection unit 540, a communication unit 550, and a power supply unit 560.

[0027] As described above, a fisheye lens with a shooting angle of view of 190 degrees is used as the shooting lens 210. Therefore, the imaging unit 10-1 can shoot an image of a range wider than a hemisphere centered on the optical axis X200. In addition, the subject side of the direction indicated by the optical axis X200 of the photographing lens 210 will be referred to as the direction of the imaging unit 10-1.

[0028] The control unit 510 is configured with a CPU, a ROM, a RAM, etc., and controls the operation of each unit of the imaging unit 10-1 based on a control program. The control unit 510 causes the imaging unit 10-1 to perform an imaging operation based on commands of the control program. The first processing unit 520 performs a first stage of signal processing on the image signal output from the imaging sensor 220 of the imaging unit 10-1. In this embodiment, as an example of the first stage of signal processing, the first processing unit 520 performs color adjustment processing, image quality adjustment processing, resolution adjustment processing, and compression processing when acquiring a moving image. The amount of communication required to transmit the image data compressed by the compression processing to the cloud 50 is reduced compared to uncompressed data.

[0029] The second processing unit 530 performs second-stage signal processing on the image signal after the first-stage signal processing. In this embodiment, as an example of the second-stage signal processing, the second processing unit 530 performs subject extraction processing, subject recognition processing, face recognition processing, and processing for improving appearance (for example, skin beautification processing for a portrait photo), etc. In addition, as the second-stage signal processing, the second processing unit 530 can also perform conversion processing for converting a hemispherical image acquired through a fisheye lens into a two-dimensional image, for example, an equirectangular image. If there is no plan to perform the second stage of signal processing, the second processing unit 530 may be omitted.

[0030] The position detection unit 540 detects the positional relationship between the two imaging units 10-1 when two camera units 10A are used in combination. Specifically, it is possible to detect geometric information indicating the distance and direction between the two imaging units 10-1, and tilt information of each imaging unit 10-1 with respect to the ground. Based on the detected geometric information and tilt information between the two imaging units 10-1, it is possible to accurately detect, for example, whether the orientations of the two imaging units 10-1 are the same or opposite. The position detection unit 540 may detect the position of the base unit 10-2 itself. If there is no plan to use two camera units 10A in combination, the position detection unit 540 of the base unit 10-2 may be omitted.

[0031] The communication unit 550 communicates with the cloud 50 or the station server 40 . The power supply unit 560 supplies necessary power to each component of the base unit 10-2 and the imaging unit 10-1 attached to the base unit 10-2. A battery 570 may be included as a power source for the power supply unit 560.

[0032] (2) Configuration of camera unit 10D The configuration of the camera unit 10D will be described with reference to the block diagram illustrated in Fig. 5. The camera unit 10D is composed of two imaging units 10-1a and 10-1b, and a base unit 10-4. The first imaging unit 10-1a includes a photographing lens 210, an imaging sensor 220, and the above-mentioned aperture, shutter mechanism, etc. (not shown). The second imaging unit 10-1b includes a photographing lens 310, an imaging sensor 320, and the above-mentioned aperture, shutter mechanism, etc. (not shown). As described above, by using a fisheye lens with a shooting angle of 190 degrees for imaging unit 10-1a, it is possible to capture an area wider than a hemisphere centered on optical axis X200. Similarly, by using a fisheye lens with a shooting angle of 190 degrees for imaging unit 10-1b, it is possible to capture an area wider than a hemisphere centered on optical axis X300. The base unit 10-4 includes a control unit 510, a first processing unit 520, a second processing unit 530, a position detection unit 540, a communication unit 550, and a power supply unit 560.

[0033] The configuration of each part in the base part 10-4 of the camera unit 10D is the same as that of the base part 10-2 of the camera unit 10A. However, the control part 510 of the base part 10-4 controls the imaging of both the first imaging part 10-1a and the second imaging part 10-1b based on the command of the control program, which is different from the control part 510 of the base part 10-2 that controls the imaging of one imaging part 10-1. Further, the first processing unit 520 of the base unit 10-4 differs from the first processing unit 520 of the base unit 10-2 in that the first processing unit 520 performs a first stage of signal processing on the image signal output from the imaging sensor 220 of the first imaging unit 10-1a and a first stage of signal processing on the image signal output from the imaging sensor 320 of the second imaging unit 10-1b.

[0034] As an example of the first-stage signal processing, the first processing unit 520 of the camera unit 10D performs color adjustment processing, image quality adjustment processing, resolution adjustment processing, compression processing when acquiring a moving image, and the like, similar to the camera unit 10A. In addition, when it is necessary to align the color, image quality, number of pixels, size, and the like between the image acquired by the first imaging unit 10-1a and the image acquired by the second imaging unit 10-1b, the first-stage signal processing may perform color adjustment processing, image quality adjustment processing, resolution adjustment processing, and the like between the two images.

[0035] In this embodiment, the first stage of signal processing is referred to as first processing α. If the contents or combination of the signal processing and adjustment processing performed as the first stage of signal processing differ between camera units, they may be referred to as first processing α-1, first processing α-2, ...

[0036] The second processing unit 530 of the camera unit 10D performs, as an example of the second-stage signal processing, the same object extraction processing, object recognition processing, face recognition processing, processing for improving appearance, conversion processing, etc. as the camera unit 10A. In addition, as the second-stage signal processing, the second processing unit 530 may perform, for example, a process for generating a display image for the user 2 to view using the terminal 200 or the head mounted display 100, etc., based on at least one of the data of the first and second images acquired by the first imaging unit 10-1a and the second imaging unit 10-1b.

[0037] The second processing unit 530 of the camera unit 10D may further perform, as second stage signal processing, shake reduction processing called electronic gimbal processing, lens distortion correction processing for correcting lens distortion caused by the optical characteristics of the photographing lens 210 and the photographing lens 310, depth calculation processing for calculating the distance (i.e.) from the camera unit 10D to the target object, and extraction processing for extracting the main subject by blurring the background. In this embodiment, the second stage signal processing is referred to as second processing β. If the content of each process or the combination of processes performed as the second stage signal processing differs between camera units, it may be referred to as second processing β-1, second processing β-2, ...

[0038] A brief description of depth calculation will be given. In the camera unit 10D, a stereo camera is configured in which the optical axis X200 of the first imaging unit 10-1a and the optical axis X300 of the second imaging unit 10-1b are arranged at a distance L (called the base length). The photographing lenses 210 and 310 are configured, for example, by an equidistant projection type fisheye lens. With this configuration, the distance from each optical axis X200, X300 on the image obtained by the imaging elements 220, 320 represents the direction of the target (subject). In addition, the difference in the position of the same target on the images obtained by the imaging elements 220, 320 represents the parallax. Therefore, the distance to the target can be obtained using the principle of triangulation based on the image obtained by the imaging element 220 and the image obtained by the imaging element 320.

[0039] The position detection section 540 of the camera unit 10D detects the positional relationship between the first imaging section 10-1a and the second imaging section 10-1b. Specifically, it is possible to detect geometric information between the first imaging section 10-1a and the second imaging section 10-1b, and inclination information of both imaging sections 10-1a and 10-1b with respect to the ground. In the camera unit 10D, the optical axis X200 of the first imaging section 10-1a and the optical axis X300 of the second imaging section 10-1b are originally designed to be almost parallel. In addition, the base line length L is designed to be, for example, 65 mm. Therefore, the position detection section 540 of the camera unit 10D detects that the optical axis X200 and the optical axis X300 are parallel and that the base line length L is 65 mm. By setting the base line length L to 65 mm, which is the same as the distance between the human eyes, a stereo image with a sense of depth close to that seen by the human eye can be obtained.

[0040] Geometric information between the first imaging unit 10-1a and the second imaging unit 10-1b can be detected using a sensor or a distance and direction detection function between wireless communication devices such as Bluetooth (trademark), and tilt information can be detected using a tilt sensor or the like. Regarding the geometric information, instead of using a sensor or a wireless communication device, the following method of recording the geometric information in advance may be adopted. That is, based on design information of the mounting portion of the first imaging unit 10-1a and the second imaging unit 10-1b on the base unit 10-4, information such as the interval (=baseline length L) between the optical axis X200 and the optical axis X300 is recorded in advance in the ROM of the control unit 510. The position detection unit 540 obtains the geometric information between the first imaging unit 10-1a and the second imaging unit 10-1b by reading out the information recorded in the control unit 510. The position detection section 540 may detect the position of the base section 10-4 itself.

[0041] The communication section 550 of the camera unit 10D communicates with the cloud 50 or the station server 40. A power supply section 560 of the camera unit 10D supplies necessary power to each section of the base section 10-4 and to the first imaging section 10-1a and the second imaging section 10-1b mounted on the base section 10-4.

[0042] (3) Configuration of Camera Units 10B and 10E Camera units 10B and 10E differ from camera unit 10D in that the orientations of the two imaging units 10-1a and 10-1b are different and information on the base line length L is not required. However, the configuration of imaging units 10-1a and 10-1b shown in the block diagram is the same as that of camera unit 10D (FIG. 5). Therefore, the configurations of camera units 10B and 10E are not shown and will be described with reference to FIG. 5.

[0043] In the camera units 10B and 10E, the first imaging section 10-1a includes a photographing lens 210 and an image sensor 220. The second imaging section 10-1b includes a photographing lens 310 and an image sensor 320. As described above, imaging units 10-1a of camera units 10B and 10E use a fisheye lens with a shooting angle of view of 190 degrees, thereby being able to shoot an area wider than a hemisphere centered on optical axis X200. Similarly, imaging unit 10-1b uses a fisheye lens with a shooting angle of view of 190 degrees, thereby being able to shoot an area wider than a hemisphere centered on optical axis X300. Furthermore, the configuration of each part in base part 10-3 of camera unit 10B and the configuration of each part in a base part (not shown) of camera unit 10E are similar to the configuration of each part in base part 10-4 of camera unit 10D.

[0044] Camera units 10B and 10E are similar to camera unit 10D in that a control unit 510 controls the imaging of both first imaging unit 10-1a and second imaging unit 10-1b based on instructions of a control program, and a first processing unit 520 performs a first stage of signal processing on an image signal output from imaging sensor 220 of first imaging unit 10-1a, and a first stage of signal processing on an image signal output from imaging sensor 320 of second imaging unit 10-1b.

[0045] Camera units 10B and 10E differ from camera unit 10D in the following respects: That is, in addition to the second-stage signal processing exemplified in the description of camera unit 10A, second processing unit 530 performs, as second-stage signal processing, a joining process (called a stitching process) that joins two semi-spherical images together to obtain the above-mentioned VR360 omnidirectional (panoramic) image, based on the data of the first and second images obtained by first imaging unit 10-1a and second imaging unit 10-1b. Here, since the photographing lens 210 and the photographing lens 310 have a field angle of 190 degrees, the imaging unit 10-1a photographs an area wider than a hemisphere centered on the optical axis X200, and the imaging unit 10-1b photographs an area wider than a hemisphere centered on the optical axis X300. Therefore, a so-called overlapping margin is ensured in the stitching process for stitching together the two hemisphere images, making it possible to combine the images with high accuracy.

[0046] As in the case of camera unit 10D, the second processing unit 530 of camera units 10B and 10E performs, as the second stage of signal processing, a shake reduction process known as electronic gimbal processing, and a lens distortion correction process for correcting lens distortion caused by the optical characteristics of the photographing lens 210 and the photographing lens 310. As described above, in this embodiment, the second stage signal processing is referred to as second processing β, but if the content of each process or the combination of processes performed as the second stage signal processing differs between camera units, it may be referred to as second processing β-1, second processing β-2, ....

[0047] <Combination of multiple camera units 10A> In this embodiment, a combination of multiple camera units 10A can be handled in the same way as camera units 10B, 10E, or 10D. 6 is a diagram showing an example of the arrangement of nine camera units 10A (10A-1 to 10A-9). Of the nine camera units 10A, camera units 10A-1 and 10A-2, which are capable of photographing different subjects located in opposite directions, can be treated in the same way as camera unit 10B (10E). Furthermore, among the nine camera units 10A, camera units 10A-2 and 10A-3 that are capable of photographing a common subject located in the same direction can be treated in the same way as camera unit 10D. In the example arrangement of FIG. 6, some of the imaging sections of camera unit 10A (10A-1 to 10A-9) are oriented in the same direction (0 degrees) and in the opposite direction (180 degrees). However, the imaging sections may be oriented in different directions, for example, by 90 degrees or 135 degrees.

[0048] A configuration example of a combination of camera unit 10A-2 and camera unit 10A-3 will be described with reference to the block diagram of Fig. 7. The first imaging section 10-1a included in camera unit 10A-2 includes a photographing lens 210 and an image sensor 220. The second imaging section 10-1b included in camera unit 10A-3 includes a photographing lens 310 and an image sensor 320. As described above, the first imaging unit 10-1a can capture an image of a range wider than a hemisphere centered on the optical axis X200, and the second imaging unit 10-1b can capture an image of a range wider than a hemisphere centered on the optical axis X300. The first base unit 10-2a included in the camera unit 10A-2 and the second base unit 10-2b included in the camera unit 10A-3 each include a control unit 510, a first processing unit 520, a second processing unit 530, a position detection unit 540, a communication unit 550, and a power supply unit 560.

[0049] 7, the configurations of the control unit 510, the first processing unit 520, the second processing unit 530, the position detection unit 540, the communication unit 550, and the power supply unit 560 are the same as those of the base unit 10-2 of the camera unit 10A described above. When two camera units 10A are used in combination, the position detection unit 540 can detect geometric information between the first imaging unit 10-1a included in the camera unit 10A-2 and the second imaging unit 10-1b included in the camera unit 10A-3, and inclination information of both imaging units 10-1a and 10-1b with respect to the ground. Based on the detected geometric information and inclination information between the two imaging units 10-1a and 10-1b, for example, it is accurately detected whether the orientations of the first imaging unit 10-1a and the second imaging unit 10-1b are the same or opposite. In addition, in order to detect the geometric information between the first imaging unit 10-1a and the second imaging unit 10-1b, in addition to a method of detecting it using a sensor or the like not shown, a method of recording the geometric information in advance in a ROM in the control unit 510 and reading it out when necessary may also be adopted.

[0050] In particular, first processing unit 520 in FIG. 7 may perform, in addition to the first stage of signal processing exemplified in the description of camera unit 10A above, an adjustment process as a first stage of signal processing to align the color, image quality, number of pixels, size, etc. between an image captured by first imaging unit 10-1a included in camera unit 10A-2 and an image captured by second imaging unit 10-1b included in camera unit 10A-3.

[0051] In this embodiment, the signal processing performed separately on the image acquired by the first imaging unit 10-1a and the image acquired by the second imaging unit 10-1b, and the adjustment processing for aligning the color tone between the image acquired by the first imaging unit 10-1a and the image acquired by the second imaging unit 10-1b are merely examples of the first stage of signal processing. Furthermore, the first stage of signal processing may be performed by either the first processing unit 520 included in the camera unit 10A-2 or the first processing unit 520 included in the camera unit 10A-3, or may be shared between the two.

[0052] In addition to the second stage of signal processing exemplified in the description of camera unit 10A, second processing unit 530 may perform, as second stage signal processing, a process of generating a display image for user 2 to view using terminal 200 or head-mounted display 100, etc., based on at least one of the data of the first and second images acquired by first imaging unit 10-1a included in camera unit 10A-2 and second imaging unit 10-1b included in camera unit 10A-3.

[0053] Furthermore, the second processing unit 530 in FIG. 7 may perform, as second-stage signal processing, shake reduction processing known as electronic gimbal processing, lens distortion correction processing for correcting lens distortion caused by the optical characteristics of the photographing lens 210 and the photographing lens 310, depth calculation processing for calculating the distance (i.e., depth) from the camera units 10A-2 and 10A-3 to the object, and extraction processing for extracting the main subject by blurring the background. The second stage of signal processing may be shared between either or both of second processing unit 530 included in camera unit 10A-2 and second processing unit 530 included in camera unit 10A-3.

[0054] The communication units 550 of both camera units 10A-2 and 10A-3 communicate between camera units 10A-2 and 10A-3, and the communication unit 550 of one of camera units 10A-2 and 10A-3 communicates with cloud 50 or station server 40 on behalf of camera units 10A-2 and 10A-3.

[0055] <Cloud configuration> (1) Server 50A (50B) The configuration of the server 50A (50B) of the cloud 50 will be described with reference to the block diagram illustrated in Fig. 8. The server 50A (50B) includes a control unit 551, a first processing unit 552, a second processing unit 553, a face detection unit 554, a face recognition unit 555, a communication unit 556, a power supply unit 557, and a storage unit 558.

[0056] The control unit 551 is configured with a CPU, a ROM, a RAM, etc., and controls the operation of each part of the server 50A (50B) based on a control program. In addition, the control unit 551 determines which entity will be responsible for the first stage signal processing of the image signal acquired by the camera unit 10, and the second stage signal processing of the image after the first stage signal processing, i.e., whether the first stage signal processing will be performed by the camera unit 10, the station server 40, the cloud 50, or whether the processing will be shared. The decision of the processing entity to be in charge of the processing is not limited to the server 50A (50B) of the cloud 50, but may be made by the camera unit 10 or the station server 40. In the following description, a case where the control unit 551 of the server 50A (50B) makes the decision will be exemplified.

[0057] The first processing unit 552 can perform the first stage of signal processing, similar to the first processing unit 520 of the camera unit 10 . In this embodiment, the first stage of signal processing for the image signal output from the imaging sensor 220 (or imaging sensor 320) of the camera unit 10 is normally performed by the first processing unit 520 in the camera unit 10 as the processing main body. However, for example, when the remaining charge of the battery 570 of the power supply unit 560 of the camera unit 10 falls below a predetermined value, or when the processing load state in the camera unit 10 is considered problematic (for example, the processing load is higher than a predetermined reference value), it is possible for the first processing unit 552 of the server 50A (50B) to perform the first stage of signal processing as the processing main body instead of the first processing unit 520 in the camera unit 10. The processing load state is determined, for example, by the occupancy rate of the process in the CPU, etc. multiplied by the processing time, and the frame rate (unit: fps) of the acquired moving image. The larger these values ​​are, the higher the processing load state is, and the smaller these values ​​are, the lower the processing load state is. Also, even if the content of the signal processing is the same, if the processing capacity of the CPU, etc. is high, the processing load state is low, and if the processing capacity of the CPU, etc. is low, the processing load state is high. The processing capacity and processing load state of the CPU and the like in the camera unit 10 are included in information about the camera unit 10, which will be described later, and are transmitted from the camera unit 10 to the cloud 50.

[0058] The second processing unit 553 can perform second-stage signal processing in the same manner as the second processing unit 530 of the camera unit 10. The second-stage signal processing may include a conversion process for converting a hemispherical image acquired through a fisheye lens into a two-dimensional image, for example, an equirectangular image. Furthermore, the second stage signal processing may include processing for adding dramatic effects such as particle effects to the image.

[0059] In this embodiment, the second stage of signal processing for the image signal captured by the imaging section 10-1 (10-1a or 10-1b) of the camera unit 10 is usually performed by the second processing section 553 in the server 50A (or 50B) as the main processing section. However, for example, when the battery 570 of the power supply section 560 of the camera unit 10 has a sufficient remaining capacity and the processing load in the camera unit 10 is low, it is possible for the second processing section 530 in the camera unit 10 to perform the second stage of signal processing as the main processing section instead of the second processing section 553 of the server 50A (or 50B).

[0060] The face detection section 554 detects the face of a human or animal based on the image captured by the imaging section 10-1 (10-1a or 10-1b) of the camera unit 10. The face recognition unit 555 verifies that the face detected by the face detection unit 554 is a specific person (for example, that the person is the person named XXX) by comparing the face detected by the face detection unit 554 with data registered (recorded) in advance in the memory unit 558. In addition to the face recognition unit, an object recognition unit, a behavior recognition unit, an abnormality recognition unit, a scene recognition unit, etc. may be provided. Each recognition unit recognizes the captured object, the behavior of the object, the abnormality of the object, and the captured scene by comparing the captured image with data registered (recorded) in advance in the storage unit 558 based on the acquired image.

[0061] The communication unit 556 communicates with the camera unit 10 or the station server 40 . The power supply unit 557 supplies necessary power to each component of the server 50A (50B). Storage unit 558 stores image data transmitted from camera unit 10, data subjected to the first stage of signal processing by first processing unit 552, data subjected to the second stage of signal processing by second processing unit 553, data registered for face recognition, etc. Recording and reading of data in storage unit 558 is controlled by control unit 551.

[0062] <Explanation of the flow chart> In the imaging system 1 (1A) described above, the image acquired by the camera unit 10 is stored in the cloud 50 after a predetermined process is performed. As described above, in this embodiment, the control unit 551 of the server 50A (50B) determines the entity that processes the image. The determined processing entity can be further changed by the determination of the control unit 551 of the server 50A (50B) or the control unit 510 of the camera unit 10. Specifically, the first stage of signal processing on the signal of the image acquired by the camera unit 10 is performed as the processing entity by the first processing unit 552 of the server 50A (50B) instead of the first processing unit 520 in the camera unit 10, and the second stage of signal processing on the image after the above-mentioned first stage of signal processing is performed as the processing entity by the second processing unit 530 in the camera unit 10 instead of the second processing unit 553 in the server 50A (50B). The change of the processing entity is not limited to the decision of the camera unit 10 or the server 50A (50B), but may be made by the decision of the station server 40. The transfer of information and image data between the camera unit 10 and the cloud 50 in the imaging system 1 will be described with reference to the flow charts shown in FIGS.

[0063] (Camera unit 10) 9 is a flowchart illustrating the flow of processing executed by the control unit 510 of the camera unit 10. The control unit 510 repeatedly executes the processing shown in FIG. Instead of repeatedly executing the program, the program may be configured to execute when a trigger is detected.

[0064] 9, the control unit 510 transmits information about the camera unit 10 from the communication unit 550 to the cloud 50, and proceeds to step S102. The information about the camera unit 10 includes information indicating the optical characteristics (focal length, angle of view, maximum F value, relative position from the mounting part to the base part to the photographing lens, distortion inherent to the optical system, aberration, type of control used in the imaging part, presence or absence of a zoom mechanism, presence or absence of an aperture, size of the camera unit 10, etc.) of the photographing lens 210 (photographing lens 310) constituting the imaging part 10-1, specifications (sensor size, number of pixels, etc.) of the imaging sensor 220 (imaging sensor 320), positional relationship between the multiple imaging parts 10-1, inclination information of both imaging parts 10-1 with respect to the ground, the state of the processing load in the camera unit 10, the heat generation state of the equipment (imaging part, base part), and the remaining amount of the battery 570 in the power supply part 560, etc. Furthermore, when the processor of the first stage signal processing and the processor of the second stage signal processing are determined by camera unit 10, each determined processor is included in the information about camera unit 10. Furthermore, when a storage unit is added to camera unit 10, recordable information indicating that an image can be recorded in camera unit 10 or no-recording information indicating that an image cannot be recorded may be included. The no-recording information indicates that the free space in the storage unit is less than a specified capacity or that the storage unit is broken. In the case of camera unit 10A, since there is only one imaging unit 10-1, information indicating the positional relationship of the multiple imaging units 10-1 may be omitted. Also, in the case of camera unit 10C, since there are four imaging units 10-1, information indicating the positional relationship of the four imaging units 10-1, inclination information of each imaging unit 10-1 with respect to the ground, etc. are included.

[0065] Among the information on the camera unit 10, the optical characteristics of the photographing lens 210 (photographing lens 310) and the specifications of the imaging sensor are required when the control unit 551 of the server 50A (50B) determines the contents of the first process (α or α-1, α-2, ...) as the first stage of signal processing, and in the determined process. In addition, information indicating the positional relationship of the multiple imaging units 10-1 is required when the control unit 551 of the server 50A (50B) determines the contents of the second process (β or β-1, β-2, ...) as the second stage of signal processing, and in the determined process. In addition, the remaining amount of the battery 570 in the power supply unit 560 and the state of the processing load in the camera unit 10 are required when the control unit 551 of the server 50A (50B) determines which entity will execute the first process and the second process, i.e., whether to execute them on the camera unit 10 or on the cloud 50. However, when the contents of the first stage signal processing and the contents of the second stage signal processing are determined on the camera unit 10 side, each processing content is determined using information related to the camera unit 10. The information about the camera unit 10 is based on the combination of the base unit and the imaging unit that constitute the camera unit 10. For example, when the imaging unit or the base unit is replaced with one having different specifications, the information about the camera unit 10 is determined by the specifications of the replaced imaging unit and base unit. In addition, a camera unit in which the base portion and the imaging portion are integrally configured may be used. In this case, the information about the camera unit 10 is determined by the specifications of the camera unit after the integral configuration.

[0066] In step S102, the control unit 510 judges whether or not the processing contents and the processing subject for the image have been received from the cloud 50. As described above, in this embodiment, the processing subjects of the first processing and the second processing and the respective processing contents are determined on the cloud 50 (server 50A (50B)) side. The processing contents for the image include the contents of the first processing (α or α-1, α-2, ...) as the first stage signal processing, the contents of the second processing (β or β-1, β-2, ...) as the second stage signal processing, and information indicating the processing subject whether the first processing and the second processing are performed by the camera unit 10 or the cloud 50. If the control unit 510 has received information indicating the processing contents and the like from the cloud 50, it makes an affirmative judgment in step S102 and proceeds to step S104, and if the control unit 510 has not received information indicating the processing contents and the like, it makes a negative judgment in step S102 and proceeds to step S110.

[0067] In step S104, the control unit 510 determines the processing content of the camera unit 10 for the image captured by the imaging unit 10-1, based on the content of the information received from the cloud 50. For example, based on the information (instructions) received from the cloud 50, the control unit 510 determines that the base unit 10-2 of the camera unit 10 will perform the first processing (α or α-1, α-2, ...) as the processing subject, and that the second processing (β or β-1, β-2, ...) will be omitted in the camera unit 10, and proceeds to step S106.

[0068] In step S106, control unit 510 determines whether the state of camera unit 10 satisfies a condition for performing processing on an image. It is assumed that a condition is set in advance in camera unit 10 such that processing is performed by camera unit 10 when the remaining charge of battery 570 in power supply unit 560 is, for example, 20 percent or more and the processing load state is a predetermined threshold or less. If the above condition is satisfied, control unit 510 makes an affirmative decision in step S106 and proceeds to step S108, and if the above condition is not satisfied, control unit 510 makes a negative decision in step S106 and proceeds to step S112.

[0069] In step S108, when the imaging unit 10-1 of the camera unit 10 acquires an image, the control unit 510 causes the first processing unit 520 of the base unit 10-2, or the first processing unit 520 and the second processing unit 530 of the base unit 10-2, to perform the processing determined in step S104 on the acquired image as the processing subject, and transmits the processed image data from the communication unit 550 to the cloud 50, thereby completing the processing shown in FIG. 9.

[0070] In step S110, which is reached after a negative decision in step S102, control unit 510 determines predetermined processing content recorded in control unit 510. For example, it is determined that first processing unit 520 of base unit 10-2 of camera unit 10 performs first processing α-1 as the processing subject, and that the processing of second processing β in camera unit 10 is omitted, and the process proceeds to step S106.

[0071] In step S112, which is reached after a negative decision in step S106, control unit 510 decides to omit the first process (α or α-1, α-2, ...) and the second process (β or β-1, β-2, ...) in camera unit 10, and proceeds to step S114. Deciding to omit the first process and the second process in camera unit 10 is essentially equivalent to changing the processing subject of the first process and the second process from camera unit 10 to cloud 50.

[0072] In step S114, when the imaging unit 10-1 of the camera unit 10 acquires an image, the control unit 510 transmits unprocessed image data from the communication unit 550 to the cloud 50 without processing the acquired image in the first processing unit 520 and the second processing unit 530 of the base unit 10-2 of the camera unit 10, and terminates the processing shown in FIG. 9.

[0073] (Cloud 50) Fig. 10 is a flowchart explaining the flow of processing executed by control unit 551 of cloud 50 (server 50A (50B)). Control unit 551 repeatedly executes the processing shown in Fig. 10 while communication is established with camera unit 10. Therefore, when information is determined or changed on the camera unit 10 side, the latest information is transmitted to cloud 50 in step S200 described later. Instead of repeatedly executing the process, the process may be configured to be executed when a trigger is generated.

[0074] In step S200 of FIG. 10, the control unit 551 receives information about the camera unit 10 from the camera unit 10, and the process proceeds to step S202. In step S202, the control unit 551 determines the processing content and processing entity for the image based on the information about the camera unit 10 received in step S200. As described above, the determined content includes the content of the first processing (α or α-1, α-2, ...) as the first stage of signal processing, the content of the second processing (β or β-1, β-2, ...) as the second stage of signal processing, and information indicating the processing entity whether the first processing and the second processing will be performed by the camera unit 10 or by the cloud 50. If the information regarding camera unit 10 includes recordable information, control unit 551 may determine the recording destination of the image in step S202. If the recording destination of the image is determined to be camera unit 10, the image data is transmitted to camera unit 10 for recording. If the recording destination of the image is determined to be cloud 50, the image data is recorded in memory unit 558. The following explanation will exemplify a case where the information regarding camera unit 10 does not include recordable information (the recording destination of the image data is determined to be memory unit 558).

[0075] There are roughly three patterns for sharing the processing load between the camera unit 10 and the cloud 50. The first pattern is that the camera unit 10 performs the first processing (α or α-1, α-2, ...) and the cloud 50 performs the second processing (β or β-1, β-2, ...). The second pattern is one in which the camera unit 10 performs the first process (α or α-1, α-2, . . . ) and the second process (β or β-1, β-2, . . . ). The third pattern is one in which no processing is performed by the camera unit 10, and the first processing (α or α-1, α-2, . . . ) and the second processing (β or β-1, β-2, . . . ) are performed by the cloud 50.

[0076] In step S204, the control unit 551 transmits the processing content and processing subject determined in step S202 from the communication unit 556 to the camera unit 10, and the process proceeds to step S206. When the processing content, the processing subject, etc. are determined and the information is changed on the cloud 50 side, the latest information is transmitted to the camera unit 10 in step S204. In step S206, the control unit 551 determines whether or not image data has been received from the camera unit 10. If image data has been received from the camera unit 10, the control unit 551 makes an affirmative decision in step S206 and proceeds to step S208, whereas if image data has not been received, the control unit 551 makes a negative decision in step S206 and ends the processing in FIG. The image data received from the camera unit 10 is provided with information indicating whether or not a first processing (α or α-1, α-2, ...) has been performed as a first stage of signal processing, and whether or not a second processing (β or β-1, β-2, ...) has been performed as a second stage of signal processing.

[0077] In step S208, the control unit 551 judges whether the first processing (α or α-1, α-2, ...) and the second processing (β or β-1, β-2, ...) have been performed on the image data. If the first processing (α or α-1, α-2, ...) and the second processing (β or β-1, β-2, ...) have been performed, the control unit 551 judges step S208 as positive and proceeds to step S210, and if the first processing (α or α-1, α-2, ...) and the second processing (β or β-1, β-2, ...) have not been performed, the control unit 551 judges step S208 as negative and proceeds to step S212.

[0078] When the process proceeds to step S210, no processing is required for the image in the cloud 50. In step S210, the control unit 551 stores the received image in the storage unit 558, and the process in FIG. If a negative decision is made in step S208, some kind of processing of the image is required in the cloud 50. In step S212, the control unit 551 decides whether or not a first processing (α or α-1, α-2, ...) has been performed on the image data. If the first processing (α or α-1, α-2, ...) has been performed, the control unit 551 makes a positive decision in step S212 and proceeds to step S214, and if the first processing (α or α-1, α-2, ...) has not been performed, the control unit 551 makes a negative decision in step S212 and proceeds to step S216.

[0079] When proceeding to step S214, the second processing (β or β-1, β-2, ...) on the image is required in the cloud 50. In step S214, the control unit 551 performs the second processing (β or β-1, β-2, ...) on the received image data in the second processing unit 553, stores the image data after the second processing (β or β-1, β-2, ...) in the storage unit 558, and ends the processing in FIG.

[0080] If a negative decision is made in step S212, a first process (α or α-1, α-2, ...) and a second process (β or β-1, β-2, ...) are required for the image in the cloud 50. In step S216, the control unit 551 performs the first process (α or α-1, α-2, ...) on the received image data in the first processing unit 552, and further performs the second process (β or β-1, β-2, ...) in the second processing unit 553, stores the image data after the first and second processes in the storage unit 558, and ends the processing in FIG.

[0081] <Determining what to do with the image> An example of a method by which the control unit 551 determines the first stage signal processing (first processing α, α-1, α-2, ...) and the second stage signal processing (second processing β, β-1, β-2, ...) of the processing contents for an image will be described with reference to FIG. 11. 11 is a table showing the subject that performs processing on an image and the first processing (α, α-1, α-2, ...) and the second processing (β, β-1, β-2, ...) of the processing contents when the camera unit 10 has two imaging sections 10. In this table, the subject that performs processing on an image and the processing contents are predetermined based on the combination of the two imaging sections 10. When the camera unit 10 is 10B, 10D or 10E, the control unit 551 appropriately refers to a table such as that in FIG. 11 to determine who will process the image and what processing will be performed. Furthermore, when a plurality of camera units 10A are combined, the control unit 551 can refer to the above table in the same manner as in the case of the camera unit 10B (10E) or the camera unit 10D. In addition, in cases where the camera unit 10 has one imaging section 10 and where the camera unit 10 has four imaging sections 10, the entity that performs the processing on the image and the processing content are defined in advance in a table, and the control section 551 refers to the table as appropriate.

[0082] 11, the optical characteristics LA of the photographing lens 210 are, for example, a 200-degree angle of view and an F-number of 2.0, and the optical characteristics LB of the photographing lens 310 are, for example, a 190-degree angle of view and an F-number of 2.8. Furthermore, the specifications SA of the image sensor 220 are, for example, 20 M pixels and a size of 1 / 1.7 type, and the specifications SB of the image sensor 320 are, for example, 40 M pixels and a size of 1 type.

[0083] When the optical characteristics of the photographing lens 210 and the photographing lens 310 differ between the imaging units 10-1a and 10-1b, or the specifications of the imaging sensor 220 and the imaging sensor 320 differ, these differences cause differences in the number of data, signal values ​​(image brightness, image color), image blur, etc., between the first image acquired by the imaging unit 10-1a and the second image acquired by the imaging unit 10-1b. Images having such differences may be inappropriate for viewing two images as a stereo image, or may be inconvenient when performing second-stage signal processing, for example, because the boundary becomes discontinuous when two images are stitched together by stitching. Therefore, the first stage signal processing (first processing α, α-1, α-2, ...) mainly performed by the first processing unit 520 of the base unit includes processing for suppressing the differences between the two images, specifically, signal processing for bringing the colors, image quality, resolution, etc. of the images closer together.

[0084] 11, the positional relationship P(10-3) of the imaging unit and the type B(10-3) of the base unit indicate geometric information between the imaging unit 10-1a and the imaging unit 10-1b, for example, whether the orientations of the imaging unit 10-1a and the imaging unit 10-1b are the same or opposite. Depending on whether the orientations of the imaging unit 10-1a and the imaging unit 10-1b are the same or opposite, the first image acquired by the imaging unit 10-1a and the second image acquired by the imaging unit 10-1b are viewed as a stereo image or as an omnidirectional image. Therefore, the second stage signal processing (second processing β, β-1, β-2, . . . ) mainly performed by the cloud 50 includes processing suited to the intended use of each of the two images.

[0085] Specifically, the second process β includes a stitching process. The second processes β-1 and β-2 include a conversion process to an equirectangular projection image and a conversion process to a stereo format. The second processes β-1 and β-2 may include electronic gimbal processing, lens distortion correction processing, depth calculation processing, etc., as necessary.

[0086] FIG. 12 is a diagram for explaining the details of the second process β including the stitching process. In the above explanation, the angle of view of the photographing lens 210 is 200 degrees and the angle of view of the photographing lens 310 is 190 degrees, and the stitching process is possible. However, even if both the photographing lenses 210 and 310 are fisheye lenses, the angle of view may not be 190 degrees as described above. In this case, if the sum of the angles of view of the photographing lens 210 and the photographing lens 310 is smaller than, for example, 370 degrees, the margin for joining the two images is not sufficient, and it becomes difficult to perform the stitching process. Therefore, the control unit 551 switches the processing contents based on the angles of view of the photographing lens 210 and the photographing lens 310 as shown in FIG. 12.

[0087] (Example 1) When the angle of view of the photographing lens 210 is 180 degrees or less, and the angle of view of the photographing lens 310 is 180 degrees or less, the sum of the angles of view of the photographing lens 210 and the photographing lens 310 is less than 370 degrees. In other words, it is difficult to obtain an all-directional (all-celestial sphere) image of VR360 by joining two images. Therefore, the control unit 551 determines that one of the following processes (i) to (iii) is to be performed by the server 50 as the second process β. (i) Raise an error flag. (ii) After combining the two images, the missing parts of the omnidirectional image are filled in with a black ring (called a black band). (iii) Pasting another image over the black bar in (ii) above (called a wipe) The wipe is a process based on the idea that it looks better than a black bar.

[0088] (Example 2) When the angle of view of the photographing lens 210 is 180 degrees to 200 degrees, or is 200 degrees or more, and the angle of view of the photographing lens 310 is less than 180 degrees, the sum of the angles of view of the photographing lens 210 and the photographing lens 310 is greater than 370 degrees. In other words, two images can be stitched together to obtain an all-directional (all-celestial sphere) image of VR360. The control unit 551 determines that the stitching process is to be performed by the server 50 as the second process β. On the other hand, if the sum of the angles of view of the photographing lens 210 and the photographing lens 310 is not greater than 370 degrees, it is difficult to join the two images together to obtain an all-directional (all-celestial sphere) image of VR 360. Therefore, the control unit 551 determines that one of the above three processes (i) to (iii) is to be performed as the second process β by the server 50.

[0089] (Example 3) When the angle of view of the photographing lens 310 is 180 degrees to 200 degrees, or is 200 degrees or more, and the angle of view of the photographing lens 210 is less than 180 degrees, the sum of the angles of view of the photographing lens 210 and the photographing lens 310 is greater than 370 degrees. In other words, two images can be stitched together to obtain an all-directional (all-celestial sphere) image of VR360. The control unit 551 determines that the stitching process is to be performed by the server 50 as the second process β. On the other hand, if the sum of the angles of view of the photographing lens 210 and the photographing lens 310 is not greater than 370 degrees, it is difficult to join the two images together to obtain an all-directional (all-celestial sphere) image of VR 360. Therefore, the control unit 551 determines that one of the above three processes (i) to (iii) is to be performed as the second process β by the server 50. (Example 4) When the angle of view of the photographing lens 210 is greater than 180 degrees and the angle of view of the photographing lens 310 is greater than 180 degrees, two images can be stitched together to obtain an all-directional (all-celestial sphere) image of VR 360. The control unit 551 determines that the stitching process is to be performed by the server 50 as the second process β.

[0090] According to the above-described first embodiment, the following advantageous effects can be obtained. (1) The camera unit 10 of the imaging system 1 having the camera unit 10 and the cloud 50 includes an imaging section 10-1 that acquires an image, a base section 10-2 that is combined with the imaging section 10-1, a first processing section 520 that is included in the base section 10-2 and performs signal processing on the image, a communication section 550 that outputs information on the imaging section 10-1 and the base section 10-2 to the cloud 50, and the communication section 550 that inputs processing contents determined by the cloud 50 from the cloud 50, and the first processing section 520 processes the processing contents input to the communication section 550. The cloud 50 of the imaging system 1 also includes a communication section 556 that inputs information on the imaging section 10-1 and the base section 10-2 from the camera unit 10, a control section 551 that determines processing contents for the image acquired by the imaging section 10-1 based on the information input to the communication section 556, and the communication section 556 that outputs the processing contents determined by the control section 551 to the camera unit 10. With this configuration, camera unit 10 can notify cloud 50 of the specifications of imaging unit 10-1 and the state of base unit 10-2 (for example, the state of the processing load in first processing unit 520). Meanwhile, control unit 551 of cloud 50 can flexibly determine the contents of signal processing for the image captured by imaging unit 10-1 based on the specifications of imaging unit 10-1 and the state of base unit 10-2, and notify camera unit 10 of the contents.

[0091] (2) The processing content input by communication unit 550 of camera unit 10 includes a processing subject that performs the processing. With this configuration, it is possible to know whether the input processing content is a process to be performed by camera unit 10 or not.

[0092] (3) The communication section 550 of the camera unit 10 outputs information indicating the positional relationship between the imaging sections 10-1a and 10-1b when multiple imaging sections 10-1a and 10-1b are combined, as information regarding the imaging section 10-1 and the base section 10-2. Since it is configured in this manner, it is possible to notify the cloud 50 of information that is useful in determining the processing content for the first image acquired by the imaging section 10-1a and the second image acquired by the imaging section 10-1b, for example, whether the orientations of the imaging sections 10-1a and 10-1b are the same or opposite.

[0093] (4) Base section 10-2 (or 10-3, 10-4) of camera unit 10 houses battery 570 as a power source for camera unit 10, and communication section 550 outputs information indicating the remaining charge of battery 570 as information relating to imaging section 10-1 and base section 10-2. With this configuration, it is possible to inform cloud 50, which determines the processing content for an image, of information that is useful for making the decision.

[0094] (5) The imaging section 10-1 of the camera unit 10 has an imaging sensor 220 and a photographing lens 210 that guides light from a subject to the imaging sensor 220, and the communication section 550 outputs information indicating the number of pixels of the imaging sensor 220 and the angle of view of the photographing lens 210 as information relating to the imaging section 10-1 and the base section 10-2. With this configuration, it is possible to inform the cloud 50, which determines the processing content for an image, of information that is useful for making the decision.

[0095] (6) The camera unit 10 of the imaging system 1 having the camera unit 10 and the cloud 50 includes an imaging section 10-1 that acquires an image, a base section 10-2 that is combined with the imaging section 10-1, a first processing section 520 that is included in the base section 10-2 and performs signal processing on the image, a communication section 550 that outputs information on the imaging section 10-1 and the base section 10-2 to the cloud 50, and the communication section 550 that inputs processing contents determined by the cloud 50 from the cloud 50, and the first processing section 520 processes the processing contents input to the communication section 550. The cloud 50 also includes a communication section 556 that inputs information on the imaging section 10-1 and the base section 10-2 from the camera unit 10, a control section 551 that determines processing contents for the image acquired by the imaging section 10-1 based on the information input to the communication section 556, and the communication section 556 that outputs the processing contents determined by the control section 551 to the camera unit 10. With this configuration, the camera unit 10 can notify the cloud 50 of the specifications of the imaging unit 10-1 and the state of the base unit 10-2 (for example, the state of the processing load in the first processing unit 520). Meanwhile, the control unit 551 of the cloud 50 can flexibly determine the contents of the signal processing for the image captured by the imaging unit 10-1 based on the specifications of the imaging unit 10-1 and the state of the base unit 10-2.

[0096] (7) The processing content determined by the control unit 551 of the cloud 50 includes the processing entity that performs the processing. With this configuration, it is possible to flexibly determine the content of the signal processing for the image captured by the imaging unit 10-1, including where the determined processing content is to be performed.

[0097] (8) In the imaging system 1, a camera unit 10 and a cloud 50 are connected via a network line. With this configuration, the camera unit 10 and the cloud 50 can be placed at separate locations.

[0098] (9) Cloud 50 of imaging system 1 having camera unit 10 and cloud 50 includes: imaging unit 10-1 for acquiring images; base unit 10-2 for combining with imaging unit 10-1; first processing unit 520 included in base unit 10-2 for performing signal processing on the image acquired by imaging unit 10-1; communication unit 556 for inputting information relating to imaging unit 10-1 and base unit 10-2 from camera unit 10; control unit 551 for determining the processing content to be performed by first processing unit 520 of camera unit 10 based on the information input to communication unit 556; and communication unit 556 for outputting the processing content determined by control unit 551 to camera unit 10. With this configuration, camera unit 10 can notify cloud 50 of the specifications of imaging unit 10-1 and the state of base unit 10-2 (for example, the state of the processing load in first processing unit 520). Meanwhile, control unit 551 of cloud 50 can flexibly determine the contents of signal processing for the image captured by imaging unit 10-1 based on the specifications of imaging unit 10-1 and the state of base unit 10-2, and notify camera unit 10 of the contents.

[0099] (10) The processing content determined by control unit 551 of cloud 50 includes a processing entity that performs the processing. With this configuration, it is possible to notify the camera unit 10 of whether the determined processing content is a process to be performed by the camera unit 10.

[0100] The following modifications are also within the scope of the present invention, and one or more of the modifications may be combined with the above-described embodiment. (Modification 1 of the first embodiment) We will explain multi-view image processing when the photographing lens 210 of the first imaging unit 10-1a and the photographing lens 310 of the second imaging unit 10-1b are, for example, a wide-angle lens (Wide) with a photographing angle of view of about 70 degrees, or a telephoto lens (Tele) with a photographing angle of view of about 10 degrees.

[0101] Fig. 13 is a diagram for explaining the second processing β-3 as an example of the second stage signal processing performed when a photographing lens other than a fisheye lens is used. In the first modification, the part surrounded by the thick line in Fig. 13, that is, the part where the photographing lens 210 of the first imaging unit 10-1a and the photographing lens 310 of the second imaging unit 10-1b are wide-angle or telephoto lenses, is referred to.

[0102] In the first modification, the control unit 551 switches the processing contents based on the angles of view of the photographing lenses 210 and 310 and the geometric information and tilt information between the image capturing units 10-1a and 10-1b as follows. It is assumed that the specifications of the image sensor 220 and the image sensor 320 are the same. (Example 1) When the photographing lens 210 is a wide-angle lens (Wide) and the photographing lens 310 is a wide-angle lens (Wide) and the optical characteristics of the two lenses match, or when the photographing lens 210 is a telephoto lens (Tele) and the photographing lens 310 is a telephoto lens (Tele) and the optical characteristics of the two lenses match, the control unit 551 determines that one of the following processes (i) to (iv) will be performed by the server 50 as the second process β-3.

[0103] (i) High-resolution composition The images acquired by the imaging units 10-1a and 10-1b are assumed to be images of the same subject under common imaging conditions. The server 50 (second processing unit 553) displaces one of the images acquired by the first imaging unit 10-1a (referred to as the first image) and the second imaging unit 10-1b (referred to as the second image) by half the pixel pitch (spacing) of the imaging sensors 220 and 320, and synthesizes the first and second images after the displacement, thereby effectively halving the pixel pitch and increasing the resolution. (ii) Stereo image generation The images acquired by the imaging units 10-1a and 10-1b are assumed to be images of the same subject under common shooting conditions. The server 50 (second processing unit 553) acquires stereo images using the image acquired by the first imaging unit 10-1a and the image acquired by the second imaging unit 10-1b as the left eye and the right eye, respectively, based on geometric information and tilt information between the first imaging unit 10-1a and the second imaging unit 10-1b.

[0104] (iii) HDR (High Dynamic Range) The images acquired by the imaging units 10-1a and 10-1b are images of the same subject captured with different exposures between the imaging units 10-1a and 10-1b. The server 50 (second processing unit 553) effectively widens the dynamic range of the image signals based on the image signals of corresponding pixel positions of the image acquired by the first imaging unit 10-1a and the image acquired by the second imaging unit 10-1b. (iv) Depth Calculation The images captured by the imaging units 10-1a and 10-1b are assumed to be images of the same subject captured under common shooting conditions. The server 50 (second processing unit 553) calculates the distance to the target object using the principle of triangulation based on the image captured by the imaging units 10-1a and 10-1b.

[0105] (Example 2) When the photographing lens 210 is a wide-angle lens (Wide) and the photographing lens 310 is a telephoto lens (Tele) and the optical characteristics of the two lenses are inconsistent, or when the photographing lens 210 is a telephoto lens (Tele) and the photographing lens 310 is a wide-angle lens (Wide) and the optical characteristics of the two lenses are inconsistent, the control unit 551 determines that the following processing is to be performed as the second processing β-3 by the server 50.

[0106] In the above (Example 2), one of the imaging units 10-1a and 10-1b, which has a telephoto lens (Tele), captures a subject included in an image captured by the other imaging unit having a wide-angle lens (Wide). The server 50 (second processing unit 553) cuts out an image of a region corresponding to an image (referred to as a telephoto image) acquired by the imaging unit having a telephoto lens (Tele) from an image (referred to as a wide-angle image) acquired by the other imaging unit having a wide-angle lens (Wide), adjusts the angle of view (referred to as "zoom" in FIG. 13), adjusts the resolution between the cut-out image and the telephoto image, and obtains stereo images for the left eye and the right eye based on geometric information and tilt information between the imaging units 10-1a and 10-1b.

[0107] According to the above-described variant example 1, even when a wide-angle lens (Wide) or a telephoto lens (Tele) is used as the shooting lens 210 of the first imaging unit 10-1a and the shooting lens 310 of the second imaging unit 10-1b, the imaging system 1 can appropriately perform the second stage of signal processing on the multi-view images captured by the multiple imaging units 10-1.

[0108] (Modification 2 of the first embodiment) A case will be described in which an infrared (IR) sensor that receives infrared light is used as the imaging sensor 220 (320) of the first imaging unit 10-1a or the second imaging unit 10-1b. In the second modification, the portion enclosed by the dashed line in FIG. 13 is referred to, that is, the portion where an infrared sensor is mounted in one of the first imaging unit 10-1a and the second imaging unit 10-1b. The angle of view of the photographing lens 210 (310) of the imaging unit equipped with an infrared sensor as an image sensor is assumed to be narrower than the angle of view of a telephoto lens (Tele).

[0109] In the second modification, the control unit 551 switches the processing contents as follows, based on the angles of view of the photographing lenses 210 and 310, and the geometric information and tilt information between the image capturing units 10-1a and 10-1b. (Example 1) When the photographing lens 210 of the first imaging unit 10-1a is a wide-angle lens (Wide) or the photographing lens 310 of the second imaging unit 10-1b is a wide-angle lens (Wide) and the imaging sensor of the other imaging unit is an infrared sensor, the control unit 551 decides to perform the following processing as the second processing β-3 on the server 50.

[0110] The server 50 (second processing unit 553) cuts out an image of a region corresponding to an image (called a thermal distribution image) acquired by an imaging unit having an infrared sensor from an image (called a wide-angle image) acquired by an imaging unit having a wide-angle lens (Wide), adjusts the angle of view (denoted as "deformation" in FIG. 13), adjusts the resolution between the cut-out image and the thermal distribution image, and then deforms at least one of the images so as to match the positional relationship of the objects in both images based on the geometric information and tilt information between the imaging units 10-1a and 10-1b. In this case, the center of the angle of view of the thermal distribution image and the center of the cut-out image coincide with each other.

[0111] (Example 2) When the photographing lens 210 of the first imaging unit 10-1a is a telephoto lens (Tele), or the photographing lens 310 of the second imaging unit 10-1b is a telephoto lens (Tele) and the imaging sensor of the other imaging unit is an infrared sensor, the control unit 551 decides to perform the following processing as the second processing β-3 on the server 50.

[0112] The server 50 (second processing unit 553) cuts out an image of an area where an image (referred to as a telephoto image) acquired by an imaging unit having a telephoto lens (Tele) and an image (referred to as a thermal distribution image) acquired by an imaging unit having an infrared sensor overlap, aligns the angles of view of the cut out telephoto image and the cut out thermal distribution image (referred to as "center shift" in FIG. 13), adjusts the resolution of both images, and deforms the images to align the positional relationship of the objects in both images based on the geometric information and tilt information between the imaging units 10-1a and 10-1b. In this case, the center of the angle of view of the thermal distribution image and the center of the cut out image do not necessarily coincide, so it is referred to as "center shift".

[0113] According to the above-described variant example 2, even when an infrared sensor is used in the first imaging unit 10-1a or the second imaging unit 10-1b, the imaging system 1 can appropriately perform the second stage of signal processing on the multi-view images captured by the multiple imaging units 10-1.

[0114] Second Embodiment <Overview of imaging system> The camera unit 10 may be provided in the vehicle 3. In the second embodiment of the invention, an imaging system 1C that monitors the inside and outside of the vehicle 3 to control the traveling of the traveling vehicle will be described. FIG. 14 is a schematic diagram showing an example of the imaging system 1C according to the second embodiment. The imaging system 1C is composed of six camera units 10A-11 to 10A-16 corresponding to the camera unit 10A in FIG. 1, and a cloud 50 (not shown). The camera units 10A-11 to 10A-13 are arranged inside the moving vehicle 3, and the camera units 10A-14 to 10A-16 are arranged outside the vehicle 3. The number of camera units 10A is not limited to six as shown in the figure and may be increased or decreased.

[0115] (1) External camera unit The right side of the drawing is the front of the vehicle 3, and the left side of the drawing is the rear. The camera unit 10A-16 is composed of an imaging unit 10-1r and a base unit 10-12, and the imaging unit 10-1r is attached to the base unit 10-12 provided at the front of the vehicle 3. The imaging unit 10-1r uses a fisheye lens with a shooting angle of 190 degrees, and captures an area wider than a hemisphere centered on the forward direction of the vehicle 3.

[0116] Camera unit 10A-15 is composed of imaging unit 10-1q and base unit 10-13, and imaging unit 10-1q is attached to base unit 10-13 provided at the rear of vehicle 3. Imaging unit 10-1q, which uses a fisheye lens with a shooting angle of 190 degrees, captures an image of an area wider than a hemisphere centered on the rear direction of vehicle 3.

[0117] Camera unit 10A-14 is composed of an imaging unit 10-1p and a base unit 10-11, and imaging unit 10-1p is attached to base unit 10-11 provided on the roof of vehicle 3. Imaging unit 10-1p, which uses a fisheye lens with a shooting angle of 190 degrees, captures an image of an area wider than a hemisphere, with the upper direction of vehicle 3 as its center.

[0118] The vehicle 3 is an autonomous vehicle in which driving control including driving, stopping, steering, etc. is performed by the driving control unit 4. An occupant of the vehicle 3 does not need to perform driving operations. When images acquired by the camera units 10A-14 to 10A-16, i.e., images of the surroundings outside the vehicle 3, are transmitted from the vehicle 3 to the cloud 50, driving control commands calculated on the cloud 50 side based on the transmitted images are transmitted to the vehicle 3. The driving control unit 4 of the vehicle 3 controls the driving of the vehicle 3 based on the driving control commands from the cloud 50.

[0119] (2) In-car camera unit The camera unit 10A-12 is composed of the imaging unit 10-1n and the base unit 10-11. The base unit 10-11 extends downward from the roof along a center pillar (not shown), and the imaging unit 10-1n is attached to the extended position. The imaging unit 10-1n uses a fisheye lens with a shooting angle of 190 degrees and shoots an image of the inside of the vehicle 3 wider than a hemisphere, centered on the front direction perpendicular to the drawing.

[0120] The camera unit 10A-11 is composed of an imaging unit 10-1m and the base unit 10-11. The imaging unit 10-1m is attached to the lower side of the base unit 10-11 and near the ceiling of the rear seat. The imaging unit 10-1m captures images of passengers (especially their faces) sitting in the rear seat.

[0121] The camera unit 10A-13 is composed of an imaging unit 10-1o and the base unit 10-11. The imaging unit 10-1o is attached to the lower side of the base unit 10-11 and near the ceiling of the front seat. The imaging unit 10-1o captures images of passengers (especially their faces) sitting in the front seats.

[0122] When images captured by camera units 10A-11 to 10A-13, i.e., images of the interior of the vehicle 3, are transmitted from the vehicle 3 to the cloud 50, the cloud 50 checks whether there are any abnormalities with the occupants of the vehicle 3 based on the transmitted images. In addition, the images of the faces of the occupants captured by camera units 10A-11 and 10A-13 are used for person identification processing. The vehicle interior space may be recorded in three dimensions based on images captured by the camera units 10A-11 to 10A-13 inside the vehicle. The three-dimensional images can be used for holding meetings inside the vehicle, video conferences with people outside the vehicle, inter-vehicle communication when traveling with multiple vehicles 3, and for crime prevention purposes.

[0123] <First and second treatments> In the second embodiment, the control unit 551 of the server 50A (50B) of the cloud 50 uses information about the camera units 10A-11 to 10A-16 when determining the contents of the first stage signal processing (first processing) and the second stage signal processing (second processing), and the entity (camera unit or cloud 50) that will perform the first processing and the entity (camera unit or cloud 50) that will perform the second processing. (1) First Processing Control unit 551 determines that a process of compressing moving images acquired by camera units 10A-11 to 10A-16 and a process of converting the moving images into equirectangular projection images are to be performed by each of camera units 10A-11 to 10A-16 as a first process α-10. As a result, first processing unit 520 of base unit 10-11, base unit 10-13 and base unit 10-12, which are base units of camera units 10A-11 to 10A-16, performs first processing α-10 on the moving images captured by camera units 10A-11 to 10A-16, respectively.

[0124] (2) Second Processing (2-1) Calculate the distance between vehicles The control unit 551 decides to perform, as the second process β-10 on the cloud 50, a process of calculating the inter-vehicle distance from another vehicle based on video images respectively acquired by the camera unit 10A-16 provided in front of the vehicle 3 and the camera unit 10A-15 provided at the rear of the vehicle 3. As a result, the second processing unit 553 of the server 50A (50B) of the cloud 50 calculates the inter-vehicle distance by, for example, counting the number of sets of white lines and blank sections of lane boundary lines on a highway. Specifically, when the lane boundary lines are composed of 8 m white lines and 12 m blank sections totaling 20 m, if there are 5 sets of white lines and blank sections between the vehicle and another vehicle, a inter-vehicle distance of 100 m can be calculated. The distance between the vehicle and another vehicle may be calculated based on video images captured by a stereo camera, or the distance between the vehicle and another vehicle may be obtained using a millimeter wave radar device or the like.

[0125] (2-2) Detecting obstacles The control unit 551 decides to perform, as the second process β-11 on the cloud 50, a process of determining obstacles, etc. located in the direction of travel of the vehicle 3 based on video images acquired by the camera unit 10A-16 provided in front of the vehicle 3 or the camera unit 10A-15 provided in the rear of the vehicle 3. This allows the second processing unit 553 of the server 50A (50B) of the cloud 50 to detect obstacles on the road and road depressions, etc. The control unit 551 can issue a driving control command to avoid the detected obstacles and road abnormalities.

[0126] (2-3) Generate operation information The control unit 551 detects operations on the operating members of the vehicle 3 based on video images acquired by the camera unit 10A-12 provided inside the vehicle 3, and determines that a process of generating corresponding operation information will be performed as a second process β-12 on the cloud 50. As a result, the second processing unit 553 of the server 50A (50B) of the cloud 50 detects an operation of an operating member or the like by an occupant of the vehicle 3. The control unit 551 can perform a process that is associated in advance with the detected operation, such as issuing a driving control command for an emergency stop.

[0127] (2-4) Create a panoramic image The control unit 551 determines, for example, to perform a second process β-13 on the cloud 50 to generate a panoramic image showing the scenery around the vehicle 3 in a 360-degree panorama based on video images acquired by the camera unit 10A-14 mounted on the roof of the vehicle 3. As a result, the second processing unit 553 of the server 50A (50B) of the cloud 50 generates a panoramic image suitable for recording a driving record or the like.

[0128] (2-5) Identify the occupants The control unit 551 determines that, for example, a process of identifying an occupant of the vehicle 3 based on moving images acquired by the camera units 10A-11 and 10A-13 is to be performed in the cloud 50 as the second process β-14. As a result, the face detection unit 554 and the face recognition unit 555 of the server 50A (50B) of the cloud 50 detect faces in the moving images and identify the person of the detected face based on the name and face preregistered in the storage unit 558. If the occupant in the vehicle is not a person preregistered, the control unit 551 can take measures such as not allowing the driving control unit 4 of the vehicle 3 to drive.

[0129] (6) Determine the event The control unit 551 decides to perform an event determination process, including detection of sudden illness of an occupant based on video images acquired by camera units 10A-11 and 10A-13, and detection of trouble inside the vehicle based on video images acquired by camera unit 10A-12, as a second process β-15 on the cloud 50. As a result, the second processing unit 553 of the server 50A (50B) of the cloud 50 detects the occurrence of a sudden illness in the vehicle 3 or the occurrence of a problem in the vehicle 3. The control unit 551, for example, confirms the safety of the occupants of the vehicle 3 via the communication unit 556, and sends a command to the traveling control unit 4 to evacuate the vehicle 3 to a safe place.

[0130] In the above description of the second embodiment, a case has been exemplified in which most of the second processing is performed by the server 50. However, it is assumed that the camera units 10A-11 to 10A-16 arranged in the vehicle 3 cannot communicate in real time between the vehicle 3 and the cloud 50 when the vehicle 3 is traveling in a tunnel or an underground facility, or depending on the communication conditions. When communication between the vehicle 3 and the cloud 50 is not possible, the second processing unit 530 of the base unit 10-11, the base unit 10-13, and the base unit 10-12, which are base units of the camera units 10A-11 to 10A-16, performs the second processing that is to be performed by the server 50 instead of the server 50. In addition, a management unit may be provided in advance to manage the second processing unit 530 of base unit 10-11, base unit 10-13, and base unit 10-12, and this management unit may manage the second processing in vehicle 3 when communication between vehicle 3 and cloud 50 is not possible.

[0131] The second processes carried out by the camera units 10A-11 to 10A-16 and the server 50 may be changed as appropriate. Then, the subject of the first process α-10 and the second processes β-10 to β-14 is changed based on the state of the processing load in the camera units 10A-11 to 10A-16. Furthermore, when real-time communication between the vehicle 3 and the cloud 50 is not possible, the camera units 10A-11 to 10A-16 take the lead in carrying out the first process α-10 and the second processes β-10 to β-14.

[0132] According to the above-described second embodiment, the following advantageous effects can be obtained. (1) Camera units 10A-16 etc. of imaging system 1C having camera units 10A-16 etc. and cloud 50 include an imaging unit 10-1r etc. that acquires images, a base unit 10-12 etc. that is combined with the imaging unit 10-1r etc., a first processing unit 520 included in the base unit 10-12 etc. that performs signal processing on the image, a communication unit 550 that outputs information regarding the imaging unit 10-1r etc. and the base unit 10-12 etc. to the cloud 50, and the communication unit 550 that inputs processing content decided by the cloud 50 from the cloud 50, and the first processing unit 520 processes the processing content input to the communication unit 550. The cloud 50 also includes a communication unit 556 that inputs information relating to the imaging units 10-1r, etc. and the base unit 10-12, etc. from the camera units 10A-16, etc., a control unit 551 that determines the processing content for images acquired by the imaging units 10-1r, etc. based on the information input to the communication unit 556, and the communication unit 556 that outputs the processing content determined by the control unit 551 to the camera units 10A-16, etc. With this configuration, the camera units 10A-16 etc. can inform the cloud 50 of the specifications of the imaging units 10-1r etc. and the state of the base unit 10-12 etc. (for example, the state of the processing load in the first processing unit 520). On the other hand, the control unit 551 of the cloud 50 can flexibly determine the contents of signal processing for images acquired by the imaging units 10-1r etc. based on the specifications of the imaging units 10A-16 etc. and the state of the base unit 10-12 etc.

[0133] (2) The processing content determined by the control unit 551 of the cloud 50 includes the processing entity that performs the processing. With this configuration, it is possible to flexibly determine the content of signal processing for images acquired by the imaging unit 10-1r, etc., including where the determined processing content is to be performed.

[0134] (3) In the imaging system 1C, camera units 10A-16 and the like and a cloud 50 are connected via a network line. With this configuration, even if the camera unit 10A-16 or the like (vehicle 3) moves to a position away from the cloud 50, information can be appropriately input and output.

[0135] (Third embodiment) <Overview of imaging system> The camera unit 10 may be installed in an office. In a third embodiment of the invention, an imaging system 1D that monitors to prevent unregistered people from entering an office will be described. FIG. 15 is a schematic diagram showing an example of the imaging system 1D according to the third embodiment. The imaging system 1D is made up of four camera units 10A-21 to 10A-24 corresponding to the camera unit 10A in FIG. 1, and a server 50A of a cloud 50. The number of camera units 10 is not limited to four as shown in the figure and may be increased or decreased, and the number of servers 50A constituting the cloud 50 is not limited to one as shown in the figure. Also, although the embodiment illustrates a state in which each camera unit 10 and server 50A are directly connected, as in the case of FIG. 1, each camera unit 10 and server 50A may be connected via a wired or wireless network communication network.

[0136] <First and second treatments> In the third embodiment, the control unit 551 of the server 50A of the cloud 50 uses information about the camera units 10A-21 to 10A-24 when determining the contents of the first process and the second process, and the entity (camera unit 10 or cloud 50) that will perform the first process and the entity (camera unit 10 or cloud 50) that will perform the second process.

[0137] In this embodiment, a process for detecting a face based on frames of a moving image acquired by camera units 10A-21 to 10A-24 is defined as a first process A. A process for identifying a person whose face is detected by first process A based on a name and face registered in advance in storage unit 558 is defined as a second process B.

[0138] Control unit 551 determines which of first process A and second process B will be performed, based on the state of the processing load in camera units 10A-21 to 10A-24, as shown in the following (Example 1) to (Example 4). The state of the processing load is determined, for example, by the occupancy rate of the process in the CPU or the like multiplied by the processing time, and the frame rate (unit: fps) of the acquired moving images. The larger these values ​​are, the higher the state of the processing load is, and the smaller these values ​​are, the lower the state of the processing load is. Furthermore, even if the content of the signal processing is the same, if the processing capacity of the CPU or the like is high, the state of the processing load is low, and if the processing capacity of the CPU or the like is low, the state of the processing load is high. The processing capacity and processing load state of the CPU etc. in each camera unit are included in the information about the corresponding camera units 10A-21 to 10A-24.

[0139] (Example 1) First and second processes are performed using the camera unit When information similar to the name and face information pre-registered in memory unit 558 is stored in second processing unit 530 and the processing load in base unit 10-21 of camera unit 10A-21 is below a first threshold, control unit 551 determines that first processing A will be performed by first processing unit 520 of base unit 10-21 and second processing B will be performed by second processing unit 530 of base unit 10-21. As a result, the first processing unit 520 detects a face based on a moving image (=referred to as an original image) acquired by the imaging unit 10-1c of the camera unit 10A-21. Data of the detected face image (=face image) is sent from the first processing unit 520 to the second processing unit 530. The second processing unit 530 identifies the person whose face is in the face image based on the name and face registered in advance. In the case of Example 1, data indicating the name (person's name) of the identified person is sent from the camera unit 10A-21 to the cloud 50.

[0140] (Example 2) The first process is performed on the camera unit, and the second process is performed on the cloud. When the processing load on the base part 10-22 of the camera unit 10A-22 is equal to or less than a second threshold which is higher than the first threshold, the control part 551 decides to perform the first processing A on the first processing part 520 of the base part 10-22 and to perform the second processing B on the server 50A (face recognition part 555) of the cloud 50. As a result, the first processing unit 520 detects a face based on a moving image (=original image) acquired by the imaging unit 10-1d of the camera unit 10A-22. Data of the detected face image (=face image) is sent from the camera unit 10A-22 to the cloud 50. The face recognition unit 555 of the server 50A identifies the person whose face is in the face image based on the name and face registered in advance in the storage unit 558.

[0141] (Example 3) First processing is performed on the camera unit, and second processing is performed on the cloud When the processing load of the base part 10-23 of the camera unit 10A-23 is equal to or less than the first threshold value, but information similar to the name and face information preregistered in the storage part 558 is not stored in the second processing part 530, the control part 551 determines that the first processing A is performed by the first processing part 520 of the base part 10-23, and the second processing B is performed by the server 50A (face recognition part 555) of the cloud 50. The operation in this case is the same as in Example 2. Therefore, the explanation of the operation of Example 3 is omitted.

[0142] (Example 4) First and second processes are performed in the cloud When the processing load of the base part 10-24 of the camera unit 10A-24 is higher than the second threshold, the control part 551 decides to perform the first process A on the server 50A of the cloud 50 (face detection part 554) and to perform the second process B on the server 50A of the cloud 50 (face recognition part 555). As a result, data of the moving image (=original image) acquired by the imaging section 10-1f of the camera unit 10A-24 is transmitted from the camera unit 10A-24 to the server 50A of the cloud 50. The face detection section 554 of the server 50A detects a face based on the original image. The face recognition section 555 of the server 50A identifies the person whose face is in the facial image based on the name and face registered in advance in the storage section 558. Note that even when a failure occurs in the processing unit of the base unit 10-24 of the camera unit 10A-24, or when a processing unit is not provided in the base unit 10-24 of the camera unit 10A-24, the control unit 551 determines that the first process A is performed by the server 50A of the cloud 50 (face detection unit 554) and the second process B is performed by the server 50A of the cloud 50 (face recognition unit 555). Error information indicating the occurrence of a failure and information indicating the device specifications, such as the absence of a processing unit, are included in the information regarding the camera unit 10A-24.

[0143] According to the above-described third embodiment, the following advantageous effects can be obtained. (1) The camera unit 10A-21 of the imaging system 1D having the camera unit 10A-21 and the server 50A includes the imaging unit 10-1c and the like for acquiring an image, the base unit 10-21 and the like combined with the imaging unit 10-1c and the like, a first processing unit 520 included in the base unit 10-21 and the like for performing a first processing A on the image, a second processing unit 530 for performing a second processing B on the image, a communication unit 550 for outputting information on the imaging unit 10-1c and the like and the base unit 10-21 and the like to the server 50A, and the communication unit 550 for inputting the processing contents of the first processing A and the second processing B determined by the server 50A from the server 50A, and the first processing unit 520 performs the first processing A of the processing contents input to the communication unit 550. The second processing unit 530 performs the second processing B of the processing contents input to the communication unit 550. The server 50A also includes a communication unit 556 that inputs information relating to the imaging unit 10-1c, etc. and the base unit 10-21, etc. from the camera unit 10A-21, etc., a control unit 551 that determines the processing contents of a first process A and the processing contents of a second process B for images acquired by the imaging unit 10-1c, etc. based on the information input to the communication unit 556, and the communication unit 556 that outputs the processing contents of the first process A and the processing contents of the second process B determined by the control unit 551 to the camera unit 10A-21, etc. With this configuration, the camera units 10A-21 etc. can inform the server 50A of the specifications of the imaging units 10-1c etc. and the states of the base units 10-21 etc. (for example, the states of the processing loads in the first processing unit 520 and the second processing unit 530). On the other hand, the control unit 551 of the server 50A can flexibly determine the processing contents of the first processing A and the second processing B for the images acquired by the imaging units 10-1c etc., based on the specifications of the imaging units 10-1c etc. and the states of the base units 10-21 etc.

[0144] (2) The processing content determined by the control unit 551 of the server 50A includes the processing entity that performs the processing. With this configuration, it is possible to flexibly determine the processing content of the first processing A and the processing content of the second processing B for the images captured by the imaging units 10-1c, etc., including whether the determined processing content is to be performed in the camera unit 10A-21, etc., or in the server 50A.

[0145] (3) In the imaging system 1D, camera units 10A-21 and the like and a server 50A are connected via a network line. With this configuration, the camera units 10A-21 and the like can be placed at positions separate from the server 50A.

[0146] (Modifications of the first to third embodiments) A configuration excluding the camera unit 10 from the above-described imaging system 1 or the like may function as an image processing system. (1) The image processing system has a second processing unit 553 that performs a second stage of signal processing using an image acquired by an imaging unit 10-1 or the like that acquires an image and that has been subjected to a first stage of signal processing by a first processing unit 520, and a control unit 551 that determines the processing content of the second processing unit 553 based on first information regarding the imaging unit 10-1 or the like and second information regarding the first processing unit 520. With this configuration, the control unit 551 can appropriately determine the processing contents of the second processing unit 553 based on, for example, the specifications of the imaging unit 10-1 and the state of the first processing unit 520, and so on.

[0147] (2) The first processing unit 520 is provided on the base unit 10-2 or the like that is combined with the imaging unit 10-1 or the like. With this configuration, the control unit 551 can appropriately determine the processing contents of the second processing unit 553 based on the state of the base unit 10-2 and the like.

[0148] (3) The control unit 551 is connected to the base unit 10-2 and the like via a network line. With this configuration, the control unit 551 and the base unit 10-2 etc. can be disposed at positions separated from each other.

[0149] (4) The second processing unit 553 is connected to the base unit 10-2 and the like via a network line. With this configuration, the second processing section 553 and the base section 10-2 etc. can be disposed at positions separated from each other.

[0150] <Program> The process executed by the control unit 551 of the server 50A of the image processing system, i.e., the process exemplified in Fig. 10, may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be supplied to the server 50A (50B). The "computer-readable recording medium" refers to a portable recording medium such as a flexible disk, a magneto-optical disk, an optical disk, or a memory card, and a storage device such as a hard disk built into a computer system. The computer system includes an OS (Operating System) and peripheral hardware, such as the exemplified server 50A (50B).

[0151] Furthermore, the "computer-readable recording medium" may include a recording medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and a recording medium that stores a program for a certain period of time, such as a volatile memory in a computer system that serves as a server or client in such a case. The above-mentioned program may be a program for realizing part of the above-mentioned functions, or may be a program that realizes the above-mentioned functions in combination with a program already recorded in the computer system.

[0152] 16 is a diagram for explaining the supply of a program to the server 50A (50B). The server 50A (50B) can be provided with the program recorded on, for example, a CD-ROM 953. The server 50A (50B) can also be provided with the program via a communication line 900. The computer 952 is a server computer that provides the above program, and stores the program in a recording medium 954 such as a hard disk. The communication line 900 is a communication line such as the Internet or personal computer communication, or a dedicated communication line. The computer 952 reads out the program from the recording medium 954 and transmits the program to the server 50A (50B) via the communication line 900. That is, the program is transmitted as a data signal by a carrier wave and transmitted via the communication line 900. The server 50A (50B) receives the transmitted program. Thus, the program can be supplied as a computer-readable computer program product in various forms, such as a recording medium or a carrier wave.

[0153] Although various embodiments and modifications have been described above, the present invention is not limited to these. Aspects in which the configurations shown in the embodiments and modifications are used in combination are also included within the scope of the present invention. Other aspects that are conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0154] 1, 1A, 1B, 1C, 1D...imaging system, 3...vehicle, 4...driving control unit, 10, 10A, 10B, 10C, 10D, 10E, 10A-1 to 10A-9, 10A-11 to 10A-16, 10A-21 to 10A-24, B-1 to B-6...Camera unit, 10-1, 10-1a, 10-1b, 10-1c to 10-1f, 10-1m to 10-1r ... Imaging unit, 10-2~10-4, 10-11~10-13, 10-21~10-24...Base section, 40…Station Server, 50…Cloud, 50A, 50B, C-1, S-1 to S-3...Servers, 100...Head-mounted display, 200...Terminal, 210, 310... photographing lens, 220, 320... image sensor, 510, 551... control unit, 520, 552...first processing unit, 530, 553...second processing unit, 540: position detection unit; 550, 556: communication unit; 554: face detection unit; 555: face recognition unit; 557, 560: power supply unit; 558: memory unit; 570: battery; α, α-1, α-2, α-10, A…first treatment, β, β-1, β-2, β-3, β-10~β-15, B…second treatment

Claims

1. A first image acquisition unit that acquires an image; A second image acquisition unit that acquires an image; a communication unit configured to transmit first information regarding a relative positional relationship between the first image acquisition unit and the second image acquisition unit to an external device; a processing unit that performs a process determined by the external device based on the first information on the first data output from the first image acquisition unit; An imaging system comprising:

2. In the imaging system according to claim 1, A base portion attachable to the first image acquisition portion, The processing unit is an imaging system disposed on either the first image acquisition unit or the base unit.

3. In the imaging system according to claim 2, The communication unit is an imaging system disposed in either the first image acquisition unit or the base unit.

4. In the imaging system according to any one of claims 1 to 3, the first image acquisition unit has an image sensor that captures an image of a subject; The communication unit transmits second information regarding the imaging element to the external device, The processing unit performs, as the processing, a processing determined by the external device based on the second information.

5. In the imaging system according to claim 4, The communication unit transmits the number of pixels of the imaging element to the external device as the second information.

6. In the imaging system according to any one of claims 1 to 5, a supply unit that supplies electricity to the first image acquisition unit, The communication unit transmits third information regarding a remaining amount of electricity of the supply unit to the external device; The processing unit performs, as the processing, a processing determined by the external device based on the third information.

7. In the imaging system according to any one of claims 1 to 6, the first image acquisition unit has an optical system that emits light from a subject, The communication unit transmits fourth information regarding the optical system to the external device, The processing unit performs, as the processing, a processing determined by the external device based on the fourth information.

8. In the imaging system according to any one of claims 1 to 7, The processing unit performs processing for generating an image based on the first data as the processing of the imaging system.

9. In the imaging system according to claim 8, The processing unit is an imaging system that adjusts image quality as the processing.

10. The imaging system according to claim 8, The processing unit is an imaging system that performs resolution adjustment as the processing.

11. In the imaging system according to any one of claims 1 to 7, The processing unit is an imaging system that performs compression of the first data as the processing.

12. In the imaging system according to any one of claims 1 to 11, The processing unit calculates a distance to a subject using the first data and the second data output from the second image acquisition unit as the processing.

13. In the imaging system according to any one of claims 1 to 11, The processing unit performs processing for combining the first data and second data output from the second image acquisition unit as the processing of the imaging system.

14. Transmitting first information regarding a relative positional relationship between a first image acquisition unit that acquires an image of the subject and a second image acquisition unit that acquires an image of the subject to an external device; A data processing method in which the external device performs processing determined based on the first information on first data output from the first image acquisition unit.

15. The data processing method according to claim 14, A data processing method in which a processing unit arranged in either the first image acquisition unit or a base unit attachable to the first image acquisition unit performs processing determined by the external device based on the first information.

16. The data processing method according to claim 15, A data processing method in which the first information is transmitted to an external device by a communication unit arranged in either the first image acquisition unit or the base unit.

17. A data processing method according to any one of claims 14 to 16, transmitting second information relating to an imaging element included in the first image acquisition unit, the imaging element capturing an image of a subject, to the external device; The data processing method further comprises the step of: performing, as the processing, a processing determined by the external device based on the second information.

18. The data processing method according to claim 17, a number of pixels of the imaging element transmitted to the external device as the second information;

19. A data processing method according to any one of claims 14 to 18, transmitting third information regarding a remaining amount of electricity of a supply unit that supplies electricity to the first image acquisition unit to the external device; The data processing method further comprises the step of: performing, as the processing, a processing determined by the external device based on the third information.

20. A data processing method according to any one of claims 14 to 19, transmitting fourth information relating to an optical system included in the first image acquisition unit that emits light from a subject to the external device; The data processing method, wherein the external device performs a process determined based on the fourth information as the process.

21. A data processing method according to any one of claims 14 to 20, The data processing method includes performing processing for generating an image based on the first data, as the processing.

22. The data processing method according to claim 21, The data processing method includes adjusting image quality as the processing.

23. The data processing method according to claim 21, The data processing method includes adjusting a resolution as the processing.

24. A data processing method according to any one of claims 14 to 20, The data processing method includes compressing the first data as the processing.

25. A data processing method according to any one of claims 14 to 24, The data processing method includes calculating a distance to a subject using the first data and second data output from the second image acquisition section as the processing.

26. A data processing method according to any one of claims 14 to 24, The data processing method includes, as the processing, processing for combining the first data and second data output from the second image acquisition section.

27. ​​A program that causes a data processing device to function as a processing unit that transmits first information regarding the relative positional relationship between a first image acquisition unit that acquires an image of a subject and a second image acquisition unit that acquires an image of the subject to an external device, and performs processing determined by the external device based on the first information on first data output from the first image acquisition unit.

Images