Imaging device, and control program

Abstract

To control correction by electronic gimbal.SOLUTION: An imaging apparatus that can be mounted on a moving body includes an imaging unit that captures an image of a subject, an acquisition unit that acquires first information on a change in the posture of the imaging apparatus, an identification unit that identifies an invisible area in which the outside of the moving body is invisible on the basis of image data of the subject generated by the output from the imaging unit, and a determination unit that executes either one of a first determination process of determining a variation parameter for varying the position of an extraction range of output target image data from the image data on the basis of the first information acquired by the acquisition unit, and a second determination process different from the first determination process on the basis of the invisible area identified by the identification unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device and a control program.

Background Art

[0002] Conventionally, there has been a camera that controls its attitude by an electronic gimbal using detection signals from an acceleration sensor or a gyro sensor and performs imaging (see, for example, Patent Document 1 below). However, in the above-described conventional technology, switching automatically between a mode of imaging using an electronic gimbal and a mode of imaging without using an electronic gimbal has not been considered.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] An imaging device according to one aspect of the invention disclosed in the present application is an imaging device mountable on a moving body, including an imaging unit that images a subject, and first information regarding a change in the attitude of the imaging device by the sensor an acquisition unit that acquires, a specifying unit that specifies a non-visible region where the outside of the moving body cannot be visually recognized based on the image data of the subject generated by an output from the imaging unit, a first determination process that determines a variation parameter for varying the position of an extraction range of output target image data from the image data based on the first information acquired by the acquisition unit, and a second determination process different from the first determination process, and a determination unit that executes one of the determination processes based on the non-visible region specified by the specifying unit second information indicating a change in the posture of the imaging device obtained from the motion vector in, the first information, and 。 comprising, the second determination process is a process of invalidating the detection signal of the sensor or a process of determining a variation parameter for varying the position of the extraction range based on the image data 。

[0005] A control program which is an aspect of the invention disclosed in the present application is a control program that can be mounted on a moving body and causes a processor to control an imaging device that images a subject. The processor is caused to obtain first information regarding a change in the posture of the imaging device, and based on the image data of the subject imaged by the imaging device, identify a non-visible area where the outside of the moving body cannot be visually recognized. Based on the information, a first determination process for determining a variation parameter for varying the position of the extraction range of the output target image data from the image data, and among a second determination process different from the first determination process, based on the non-visible area, cause one of the determination processes to be executed. by the sensor obtain, and based on the image data of the subject imaged by the imaging device, identify a non-visible area where the outside of the moving body cannot be visually recognized, and First a first determination process for determining a variation parameter for varying the position of the extraction range of the output target image data from the image data based on the information, and among a second determination process different from the first determination process, based on the non-visible area second information indicating a change in the posture of the imaging device obtained from the motion vector in, the first information, and execute one of the determination processes. the second determination process is a process of invalidating the detection signal of the sensor or a process of determining a variation parameter for varying the position of the extraction range based on the image data .

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

DETAILED DESCRIPTION OF THE INVENTION

[0007] The imaging device according to this embodiment can be mounted, for example, on a moving body or a passenger of a moving body, and has an omnidirectional camera (also referred to as a 360-degree camera) that images a subject and captures omnidirectional image data. Such an imaging device is equipped with an electronic gimbal. This electronic gimbal detects the inclination of the imaging device and corrects the position of the extraction range for extracting output target image data from the omnidirectional image data so as to cancel out the inclination.

[0008] When the imaging device is mounted on a moving body or a passenger of a moving body, acceleration is applied to the imaging device due to the acceleration, deceleration, or turning of the moving body. As a result, the electronic gimbal corrects the captured image data as if the imaging device is tilted, even though the imaging device is not actually tilted. In this embodiment, by controlling so as not to perform such correction by the electronic gimbal, based on the image data outside the moving body where the subject is not the moving body, an incorrect position change of the extraction range for extracting the output target image data due to the acceleration, deceleration, or turning of the moving body is suppressed. Thereby, image data suitable for the actual running of the moving body can be provided.

[0009] Hereinafter, this embodiment will be described in detail. Examples of the moving body include vehicles (e.g., automobiles, motorcycles, bicycles, trains), ships, flying objects (e.g., airplanes, helicopters, drones), and robots. In the following embodiments, an automobile will be described as an example of the moving body. When the moving body is an automobile, the imaging device is detachably fixed inside the vehicle or to the passenger. In the following embodiments, the case where it is fixed at an arbitrary position inside the vehicle will be described as an example.

[0010] In addition, the omnidirectional camera is a camera in which the back sides of two imaging elements face each other, and a fisheye lens with a viewing angle of 180 degrees or more is provided in front of each imaging element. It is assumed that the optical axes of both fisheye lenses coincide. The omnidirectional camera generates omnidirectional image data that can rotate 360 degrees in any direction around the imaging device by synthesizing the image data from both imaging elements. The omnidirectional camera is equipped with an electronic gimbal. The electronic gimbal detects the tilt of the omnidirectional camera using an acceleration sensor and a gyro sensor, and corrects the image data so as to cancel out the tilt.

[0011] Note that the omnidirectional camera may have a configuration in which imaging elements and lenses are provided at six locations: up, down, front, back, left, and right. Also, instead of the omnidirectional camera, a camera equipped with one imaging element and one fisheye lens may be used. Further, instead of the fisheye lens, an ultra-wide-angle lens with a viewing angle of, for example, 170° or more and less than 180° may be used. Hereinafter, the embodiment will be described.

[0012] <Imaging Examples by an Imaging Device Mounted Inside a Vehicle> FIGS. 1 to 3 are explanatory diagrams showing Imaging Examples 1 to 3 by an imaging device mounted inside a vehicle. FIG. 1 is Imaging Example 1 when the vehicle 100 accelerates, FIG. 2 is Imaging Example 2 when the vehicle 100 decelerates, and FIG. 3 is Imaging Example 3 when the vehicle 100 turns to the right in the traveling direction. The imaging example when the vehicle 100 turns to the left in the traveling direction is omitted from the explanation because it is a turn in the opposite direction to Imaging Example 3. Note that the omnidirectional spherical camera which is the imaging device 101 is fixed inside the vehicle 100 with the imaging elements and the fish-eye lenses 102a and 102b facing the traveling direction and the reverse direction of the vehicle 100, respectively.

[0013] In FIG. 1, (A) shows the types of acceleration applied to the imaging device 101 when the vehicle 100 accelerates. The optical axes OA of the first fish-eye lens 102a on the traveling direction side and the second fish-eye lens 102b on the reverse direction side coincide. The imaging direction CD1 indicates the direction in which the imaging element and the first fish-eye lens 102a on the traveling direction side of the direction of the optical axis OA are facing. When the vehicle 100 accelerates, the imaging device 101 is applied with the gravitational acceleration g in the vertical direction and the acceleration a1 in the reverse direction of the vehicle 100. The combined acceleration ga1 is the acceleration obtained by combining the gravitational acceleration g and the acceleration a1.

[0014] (B) shows an example of tilt detection of the imaging device 101 when the vehicle 100 accelerates. The electronic gimbal of the imaging device 101 determines that the traveling direction side of the imaging device 101 is tilted upward in the pitch direction (tilt angle α) due to the combined acceleration ga1 (however, actually the imaging device 101 is fixed and does not tilt). The electronic gimbal changes the imaging direction CD1 to the imaging direction CD2 so that the first fish-eye lens 102a on the traveling direction side faces the front in the traveling direction, that is, so as to cancel the tilt angle α. Note that since the imaging device 101 is fixed, actually the optical axis OA does not shift like the imaging direction CD2.

[0015] (C) shows an example of the output of image data of a subject by the imaging device 101 when the acceleration a1 is not applied to the imaging device 101. Specifically, for example, (C) shows the extraction range 110 of the omnidirectional image data and the output target image data 111 extracted from the omnidirectional image data within the extraction range 110, when the imaging direction CD1 is set as the projection direction of the stitched image data. The omnidirectional image data is the image data obtained by pasting the stitched image data onto a virtual spherical surface. Among the output target image data 111, the image data 111a is the image data of the scenery outside the vehicle captured through the windshield, and the image data 111b is the image data inside the vehicle.

[0016] (D) shows an example of the output of image data of a subject by the imaging device 101 when the acceleration a1 is applied to the imaging device 101. Specifically, for example, (D) shows the extraction range 120 of the omnidirectional image data and the output target image data 121 extracted from the omnidirectional image data within the extraction range 120, when the imaging direction CD2 is set as the projection direction of the stitched omnidirectional image data. Among the output target image data 121, the image data 121a is the image data of the scenery outside the vehicle captured through the windshield, and the image data 121b is the image data inside the vehicle.

[0017] As shown in (D), each time the acceleration a1 is applied to the imaging device 101, the extraction range 110 is changed to the extraction range 120, so that the output target image data 111 is blurred to the output target image data 121. Therefore, at least a part of the region that should be originally output (in this example, the image data 121a of the scenery outside the vehicle) is not output because it is outside the extraction range 110, and an extra region (in this example, the image data 121b inside the vehicle) is included in the extraction range 110.

[0018] In FIG. 2, (A) shows the types of acceleration applied to the imaging device 101 when the automobile 100 decelerates. When the automobile 100 decelerates, the imaging device 101 is applied with the gravitational acceleration g in the vertical direction and the acceleration (negative acceleration) a2 in the traveling direction of the automobile 100. The combined acceleration ga2 is the acceleration obtained by combining the gravitational acceleration g and the acceleration a2.

[0019] (B) shows an example of tilt detection of the imaging device 101 when the vehicle 100 decelerates. The electronic gimbal of the imaging device 101 determines that the forward direction side of the imaging device 101 is tilted downward in the pitch direction (tilt angle α) based on the combined acceleration ga2 (however, actually the imaging device 101 is fixed and does not tilt). The electronic gimbal changes the imaging direction CD1 to the imaging direction CD3 so that the first fisheye lens 102a on the forward direction side faces the front in the forward direction, that is, to cancel the tilt angle α. Since the imaging device 101 is fixed, actually the optical axis OA does not shift like the imaging direction CD2.

[0020] (C) shows an example of imaging a subject by the imaging device 101 when the acceleration a2 is not applied to the imaging device 101. Specifically, for example, (C) shows the extraction range 110 of the omnidirectional image data and the output target image data 111 extracted from the omnidirectional image data within the extraction range 110 when the imaging direction CD1 is set as the projection direction of the stitched omnidirectional image data, similar to (C) in FIG. 1.

[0021] (D) shows an example of imaging a subject by the imaging device 101 when the acceleration a2 is applied to the imaging device 101. Specifically, for example, (D) shows the extraction range 130 of the omnidirectional image data and the output target image data 131 extracted from the omnidirectional image data within the extraction range 130 when the imaging direction CD3 is set as the projection direction of the stitched omnidirectional image data. The output target image data 131 is image data of the scenery outside the vehicle captured through the windshield.

[0022] As shown in (D), each time the acceleration a2 is applied to the imaging device 101, the extraction range 110 is changed to the extraction range 130, so the output target image data 111 blurs to the output target image data 131, and the visibility decreases.

[0023] In FIG. 3, (A) shows the types of acceleration applied to the imaging device 101 when the vehicle 100 turns to the right. When the vehicle 100 turns to the right, the imaging device 101 is subjected to the gravitational acceleration g in the vertical direction and the acceleration in the turning direction of the vehicle 100 (centripetal acceleration). The combined acceleration ga3 is the acceleration obtained by combining the gravitational acceleration g and the centripetal acceleration.

[0024] (B) shows an example of tilt detection of the imaging device 101 when the vehicle 100 turns to the right. The electronic gimbal of the imaging device 101 determines that the advancing direction side of the imaging device 101 is tilted in the right yaw direction (tilt angle α) based on the combined acceleration ga3 (however, in reality, since the imaging device 101 is fixed, it does not tilt). The electronic gimbal changes the imaging direction CD1 to the imaging direction CD4 so that the first fisheye lens 102a on the advancing direction side faces the front in the advancing direction, that is, to cancel the tilt angle α. Since the imaging device 101 is fixed, in reality, the optical axis OA does not shift like the imaging direction CD4.

[0025] (C) shows an example of imaging a subject by the imaging device 101 when the centripetal acceleration is not applied to the imaging device 101. Specifically, for example, (C) shows the extraction range 110 of the omnidirectional image data and the output target image data 111 extracted from the omnidirectional image data within the extraction range 110 when the imaging direction CD1 is the projection direction of the stitched omnidirectional image data, similar to (C) in FIG. 1.

[0026] (D) shows an example of imaging a subject by the imaging device 101 when the centripetal acceleration is applied to the imaging device 101. Specifically, for example, (D) shows the extraction range 140 of the omnidirectional image data and the output target image data 141 extracted from the omnidirectional image data within the extraction range 130 when the imaging direction CD4 is the projection direction of the stitched omnidirectional image data. Among the output target image data 141, the image data 141a is the image data of the scenery outside the vehicle imaged through the windshield, and the image data 141b is the image data inside the vehicle.

[0027] As shown in (D), each time the acceleration a2 is applied to the imaging device 101, the extraction range 110 is changed to the extraction range 130, so that the output target image data 111 blurs into the output target image data 131, and the visibility decreases.

[0028] Thus, in order to suppress the above-described decrease in visibility, the imaging device 101 according to this embodiment implements two modes that can be switched manually or automatically. One is the sensor usage mode in which, as usual, tilt detection is performed by an electronic gimbal using detection signals from an acceleration sensor and a gyro sensor.

[0029] When the sensor usage mode is applied, whether the imaging device 101 is mounted on the vehicle 100 or not, the electronic gimbal performs tilt detection and changes the extraction range set according to the projection direction in accordance with the detected tilt. Therefore, when the imaging device 101 is mounted on the vehicle 100, the extraction range and the output target image data are changed from (C) to (D) in FIGS. 1 to 3 by the tilt detection of the electronic gimbal.

[0030] The other mode is a mode different from the sensor usage mode, that is, a suppression mode for suppressing the tilt correction by the sensor. Specifically, the suppression mode includes, for example, three modes: a stop mode, a non-use mode, and an image detection mode. The stop mode is a mode in which the output of the detection signal from the acceleration sensor or the gyro sensor is stopped. The non-use mode is a mode in which the detection signal from the acceleration sensor or the gyro sensor is received but not used. Thereby, in the stop mode or the non-use mode, since the electronic gimbal does not use the detection signal from the acceleration sensor or the gyro sensor, the extraction range of the above-described output target image data is not changed.

[0031] There is an image detection mode that detects the movement inside the vehicle from image data without correcting based on the detection signal from the acceleration sensor or the gyro sensor. In the image detection mode, the source image data is not omnidirectional image data, but the stitched image data before spherical pasting. Since the movement inside the vehicle is minute, even if inclination occurs, it does not affect visibility compared to the sensor usage mode.

[0032] Thus, when mounting the imaging device 101 inside the automobile 100, the user may set it to the suppression mode in advance so that the extraction range 110 is not changed, or the imaging device 101 automatically switches to the suppression mode when inclination is detected. Thereby, even when the electronic gimbal detects inclination, the extraction range 110 can be maintained during playback, and the visibility of the image data captured during driving in the automobile 100 can be improved.

[0033] <Hardware Configuration Example of Imaging Device 101> FIG. 4 is a block diagram showing a hardware configuration example of the imaging device 101. The imaging device 101 includes a processor 401, a storage device 402, an operation device 403, a sensor 404, an LSI (Large Scale Integration) 405, an omnidirectional camera 406, and a communication IF (Interface) 407. These are connected by a bus 408. The processor 401 controls the imaging device 101. The storage device 402 serves as the working area of the processor 401.

[0034] The storage device 402 is a non-temporary or temporary recording medium that stores various programs and data. Examples of the storage device 402 include a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), and a flash memory. The operation device 403 operates data. Examples of the operation device 403 include a button and a touch panel.

[0035] The sensor 404 is an acceleration sensor or a gyro sensor for generating information regarding the attitude of the imaging device 101. The acceleration sensor is the sensor 404 that detects acceleration. In the XYZ coordinate system spanned by the X-axis, Y-axis, and Z-axis (the origin O is the center of gravity of the imaging device 101), for example, the X-axis is the traveling direction of the vehicle 100, the Z-axis is the vertical direction, and the Y-axis is parallel to the X-axis and the Z-axis. In this case, the acceleration sensor detects an acceleration v = (vx, vy, vz). vx is the acceleration in the X-axis direction, vy is the acceleration in the Y-axis direction, and vz is the acceleration in the Z-axis direction.

[0036] The gyro sensor is the sensor 404 that detects an angular velocity ω = (ωy, ωp, ωr). ωy is the angular velocity of the yaw angle θy around the Z-axis, ωp is the angular velocity of the pitch angle θp around the Y-axis, and ωr is the angular velocity of the roll angle θr around the X-axis. The acceleration v is converted into the angular velocity ω by the processor 401 or the LSI 405.

[0037] The LSI 405 is an integrated circuit that executes specific processes such as image processing and compression / expansion processing. The omnidirectional camera 406 images a subject and generates omnidirectional image data. The omnidirectional camera 406 is a camera in which the back sides of two imaging elements face each other, and fisheye lenses 102a and 102b each having an angle of view of 180 degrees or more are provided in front of each imaging element. The communication IF 407 is connected to the terminal 400 via a network and transmits and receives data. The terminal 400 can receive, store, and reproduce the omnidirectional image data output from the imaging device 101.

[0038] <Functional configuration example of the imaging device 101> FIG. 5 is a block diagram showing a functional configuration example of the imaging device 101. The imaging device 101 includes an imaging unit 501, an acquisition unit 502, a specifying unit 503, a determination unit 504, a motion detection unit 505, a luminance detection unit 506, a distance detection unit 507, and a calculation unit 508.

[0039] The imaging unit 501 images a subject inside the vehicle 100. Specifically, the imaging unit 501 is, for example, the omnidirectional spherical camera 406 shown in FIG. 4. The imaging unit 501 includes the above-described first fisheye lens 102a, the second fisheye lens 102b, the first imaging element 501a, the second imaging element 501b, and the image processing unit 501c. The first fisheye lens 102a condenses the light from the first subject in front of it and emits it to the first imaging element 501a. The second fisheye lens 102b condenses the light from the second subject in front of it and emits it to the second imaging element 501b.

[0040] The first imaging element 501a receives the light of the first subject incident through the first fisheye lens 102a, performs photoelectric conversion, and generates first circumferential fisheye image data. In the case of this example, the first circumferential fisheye image data is the image data of a circumferential fisheye obtained by imaging the first subject as a subject on the traveling direction side from inside the vehicle. The second imaging element 501b receives the light of the second subject incident through the second fisheye lens 102b, performs photoelectric conversion, and generates second circumferential fisheye image data. In the case of this example, the second circumferential fisheye image data is the image data of a circumferential fisheye obtained by imaging the second subject as a subject on the traveling direction side from inside the vehicle.

[0041] The image processing unit 501c converts the first circumferential fisheye image data and the second circumferential fisheye image data into first orthographic cylindrical conversion image data and second orthographic cylindrical conversion image data, respectively, divides the second orthographic cylindrical conversion image data, combines (stitches) it with the first orthographic cylindrical conversion image data, and outputs combined orthographic cylindrical conversion image data (see FIG. 8).

[0042] The image processing unit 501c generates omnidirectional image data that can rotate 360 degrees in any direction with the center of the virtual sphere corresponding to the center of gravity of the imaging device 101 as the rotation center by pasting the combined orthographic cylindrical conversion image data onto the virtual spherical surface of the virtual sphere. The image processing unit 501c stores the combined orthographic cylindrical conversion image data and the omnidirectional image data in the storage device 402.

[0043] Note that the image processing unit 501c is realized by causing the processor 401 to execute a program stored in the storage device 402 shown in FIG. 4, or by the LSI 405.

[0044] FIG. 6 is an explanatory diagram showing an example (first half) of generating omnidirectional image data by the image processing unit 501c. In FIG. 6, (A) shows the imaging state of the subject image by the first fisheye lens 102a and the second fisheye lens 102b. (B) shows the first circumferential fisheye image data 601a and the second circumferential fisheye image data 601b that are input from the first image sensor 501a and the second image sensor 501b and are subjected to debayer processing by the imaging in (A). (C) shows the orthographic cylindrical transformation. The first orthographic cylindrical transformation image data 602a and the second orthographic cylindrical transformation image data 602b are image data orthographically cylindrically transformed from the first circumferential fisheye image data 601a and the second circumferential fisheye image data 601b, respectively.

[0045] FIG. 7 is an explanatory diagram showing an example (second half) of generating omnidirectional image data by the image processing unit 501c. In FIG. 7, (D) shows post-image division, that is, the divided image data 602b1 and 602b2 obtained by dividing the second orthographic cylindrical transformation image data 602b in (C). (E) shows stitching. The combined orthographic cylindrical transformation image data 701 is image data obtained by stitching the first circumferential fisheye image data 601a and the two divided image data 602b1 and 602b2.

[0046] (F) shows the process of attaching the combined orthographic cylindrical transformation image data 701 to the virtual spherical surface 703 of the virtual sphere 702. (G) shows the parallel projection onto the virtual spherical surface 703. The combined orthographic cylindrical transformation image data 701 is attached to the virtual spherical surface 703, and the output target image data 710 is extracted from the virtual spherical surface 703 within the extraction range TA by parallel projection from the projection direction (for example, the direction of the optical axis OA). The extraction range TA in (G) is set at the time of reproducing the combined orthographic cylindrical transformation image data 701 after (F).

[0047] Returning to FIG. 5, the acquisition unit 502 acquires first information regarding changes in the posture of the imaging device 101. Specifically, for example, the acquisition unit 502 acquires the first information based on the detection signal from the sensor 404. When the sensor 404 is a gyro sensor, since the detection signal is an angular velocity, the acquisition unit 502 acquires the detection signal of the angular velocity as the first information. When the sensor 404 is an acceleration sensor, since the detection signal is an acceleration, the acquisition unit 502 converts the acceleration into an angular velocity.

[0048] Specifically, for example, the acquisition unit 502 integrates the acceleration and calculates the amount of movement per unit time (x2 - x1, y2 - y1, z2 - z1) in the XYZ coordinate system. The coordinate values (x1, y1, z1) are the positions before movement, and the coordinate values (x2, y2, z2) are the positions before movement. The acquisition unit 502 determines the movement angle from the coordinate value (x1, y1, z1) to the coordinate value (x2, y2, z2) based on the coordinate values (x1, y1, z1), the coordinate values (x2, y2, z2), and the origin O, and differentiates it per unit time to calculate the angular velocity (ωy, ωp, ωr) as the first information per unit time.

[0049] Note that the acquisition unit 502 is specifically realized, for example, by causing the processor 401 to execute a program stored in the storage device 402 or by the LSI 405.

[0050] The specific unit 503 identifies a non-visible area where it is impossible to visually recognize the outside of the moving body based on the image data of the subject generated by the output from the imaging unit 501. The image data referred to here is the combined ortho-rectangular cylindrical conversion image data 701 in Fig. 7(E). Specifically, the non-visible area is an area where it is impossible to visually recognize anything other than the moving body on which the imaging device 101 is mounted. The moving body includes the interior space of the moving body such as the driver's seat, passenger seat, and rear seat of the moving body, and the outer surface of the moving body that can be visually recognized from the interior space (for example, the hood of an automobile). In other words, the non-visible area where it is impossible to visually recognize the outside of the moving body is an area that is shielded by the interior space of the moving body and the outer surface of the moving body, and where the scenery outside the moving body cannot be visually recognized from the inside of the moving body. On the other hand, the visible area is an area outside the moving body (for example, scenery) that can be visually recognized from the interior space of the moving body on which the imaging device 101 is mounted.

[0051] Fig. 8 is an explanatory diagram showing the visible area and non-visible area in the combined ortho-rectangular cylindrical conversion image data 701. For example, there are the inner peripheral surface that becomes the interior space of the vehicle, the back surface of the door panel, the dashboard, the armrest, the center console, the console box, the glove box, the instrument panel, the seat, and the like. Also, the hood visible through the windshield is also a non-visible area. On the other hand, the scenery outside the vehicle visible through the windshield is a visible area. The window of the door is a visible area because the scenery outside the vehicle can be visually recognized regardless of whether the window glass is open or closed.

[0052] Fig. 9 is an explanatory diagram showing the motion vectors of the visible area in the combined ortho-rectangular cylindrical conversion image data 701. If the automobile 100 is in motion, the scenery in the visible area changes, but the non-visible area shows the same scenery. Therefore, the specific unit 503 detects the motion vector mv from the group of time-series combined ortho-rectangular cylindrical conversion image data 701, identifies the area where the motion vector mv is equal to or greater than a predetermined magnitude as the visible area, and identifies the area where the motion vector mv is less than the predetermined magnitude as the non-visible area.

[0053] Note that the specific part 503 is realized, for example, by causing the processor 401 to execute a program stored in the storage device 402 or by the LSI 405.

[0054] The determination unit 504 executes one of the first determination process and the second determination process different from the first determination process based on the invisible region specified by the specifying unit 503. The first determination process is a process of determining a variation parameter for varying the position of the extraction range TA of the output target image data 710 from the image data based on the first information acquired by the acquisition unit 502.

[0055] Specifically, for example, the first determination process is a process according to the above-described sensor usage mode. The determination unit 504 integrates the angular velocity, which is the first information, to obtain an angle. The direction of the angular velocity and the obtained angle serve as variation parameters for varying the position of the extraction range TA of the output target image data 710. The acquisition unit 502 and the first determination process of the determination unit 504 function as an electronic gimbal.

[0056] FIG. 10 is an explanatory diagram showing the amount of movement of the position of the extraction range TA. The projection line PD1 is a line segment passing through the center of gravity of the extraction range TA1 in the combined orthographic cylindrical conversion image data 701 and is the projection direction in (G) of FIG. 7. α is the angle obtained by integrating the angular velocity, which is the first information, that is, the amount of variation regarding the position of the extraction range TA1. The angle α, which is the amount of variation, together with the direction of the angular velocity, serves as a variation parameter. The projection line PD2 is a line segment obtained by transitioning the projection line PD1 by an angle α from the origin O in the direction of the angular velocity, and thereby the extraction range TA1 moves to become the extraction range TA2. The projection line PD2 passes through the center of gravity of the extraction range TA2.

[0057] Returning to FIG. 5, the second determination process of the determination unit 504 is a mode different from the sensor usage mode, that is, a suppression mode for suppressing the inclination correction by the sensor 404. More specifically, for example, a stop mode for stopping the output of the detection signal from the acceleration sensor or the gyro sensor, a non-use mode for not using the detection signal, and an image detection mode for determining a variation parameter for varying the position of the extraction range TA based on the combined ortho-rectangular cylindrical conversion image data after stitching in FIG. 7(E).

[0058] In the image detection mode, the source image data for detection is not the omnidirectional image data but the combined ortho-rectangular cylindrical conversion image data 701 after stitching in FIG. 7(E). Since the movement inside the vehicle, that is, the variation amount (angle) of the variation parameter is minute, even if an inclination occurs, it does not affect the variation of the position of the extraction range TA compared to the sensor usage mode.

[0059] Specifically, the determination unit 504 is realized, for example, by causing the processor 401 to execute a program stored in the storage device 402 or by the LSI 405.

[0060] The motion detection unit 505 detects a motion vector mv based on the time-series image data group generated from the output of the imaging unit 501. This image data is the combined ortho-rectangular cylindrical conversion image data 701 after stitching in FIG. 7(E). Specifically, for example, the motion detection unit 505 detects the motion vector mv based on the movement angle of the region of interest in two consecutive combined ortho-rectangular cylindrical conversion image data in terms of time.

[0061] FIG. 11 is an explanatory diagram showing the movement angle obtained by the motion detection unit 505. Point P1 is a point on the attention area of the preceding combined orthographic cylindrical conversion image data among two temporally consecutive combined orthographic cylindrical conversion image data, and point P2 is a point on the attention area of the subsequent combined orthographic cylindrical conversion image data. That is, it shows that point P1 has moved to point P2. In the orthographic cylindrical coordinate system, point P1 is a point specified by a radius r, a yaw angle θ1, and a pitch angle φ1, and point P2 is a point specified by a radius r, a yaw angle θ2, and a pitch angle φ2.

[0062] The radius r is the subject distance to a subject inside the vehicle that becomes an invisible area (for example, the inner peripheral wall of the vehicle interior space). The movement angle α is the angle formed by line segment OP1 and line segment OP2. Also, the vector passing through the virtual spherical surface 703 with point P1 as the starting point and point P2 as the ending point is the movement vector mv.

[0063] In this case, the specifying unit 503 specifies the above-described invisible area based on this movement vector mv. For example, when both the direction and magnitude differences of each movement vector mv in the movement vector mv group are within the allowable range, the specifying unit 503 specifies the visible area and the invisible area for each area where the movement vector mv group exists. Specifically, for example, if the direction and magnitude of each movement vector mv are the same and the magnitude of each movement vector mv is equal to or greater than the threshold value, the area where the movement vector mv group exists is specified as the visible area.

[0064] That is, the specifying unit 503 specifies that it is an area that reflects the state where the scenery seen through the windshield or the window glass is changing. Also, if the directions of the movement vectors mv are different and the magnitude is less than the threshold value, the area where the movement vector mv group exists is specified as the invisible area. That is, the specifying unit 503 specifies that it is an area that reflects the state of the vehicle interior that hardly changes.

[0065] Note that the motion detection unit 505 may detect the above-described motion vector mv using the detection signal from the sensor 404 (angular velocity if it is a gyro sensor, acceleration if it is an acceleration sensor). If it is a detection signal from a gyro sensor, the motion detection unit 505 can obtain the moving angle by dividing the angular velocity by the frame rate of the combined ortho-azimuthal cylindrical conversion image data 701. The direction of the angular velocity and the moving angle are detected as the motion vector mv. In the case of an acceleration sensor, the motion detection unit 505 converts it into the direction of the angular velocity and the moving angle by the same polar coordinate conversion as the acquisition unit 502.

[0066] When obtaining the direction of the angular velocity and the moving angle using the combined ortho-azimuthal cylindrical conversion image data 701 and the gyro sensor, conversion from the XYZ coordinate system to the polar coordinate system becomes unnecessary, so higher accuracy can be achieved than when obtaining from an acceleration sensor.

[0067] Note that the motion detection unit 505 is specifically realized, for example, by causing the processor 401 to execute a program stored in the storage device 402 or by the LSI 405.

[0068] The luminance detection unit 506 detects the luminance of the combined ortho-azimuthal cylindrical conversion image data 701. In the imaging state by the imaging device 101, it is considered that the outside of the vehicle has a higher luminance than the inside of the vehicle. Therefore, the luminance detection unit 506 detects the luminance of the combined ortho-azimuthal cylindrical conversion image data 701 in order to divide the combined ortho-azimuthal cylindrical conversion image data 701 into a visible region and an invisible region.

[0069] In this case, the specifying unit 503 specifies the invisible region based on the luminance detected by the luminance detection unit 506. Specifically, for example, the specifying unit 503 divides the combined ortho-azimuthal cylindrical conversion image data 701, detects the luminance for each divided region, and obtains the average luminance. Then, the specifying unit 503 specifies the region with a luminance lower than the average luminance as the invisible region and the region with a luminance equal to or higher than the average luminance as the visible region.

[0070] Note that instead of the average luminance, a threshold luminance may be set in advance, and the specific part 503 may specify that a region with a luminance lower than the threshold luminance is an invisible region and a region with a luminance equal to or higher than the threshold luminance is a visible region.

[0071] Note that the luminance detection unit 506 is specifically realized, for example, by causing the processor 401 to execute a program stored in the storage device 402 or by the LSI 405.

[0072] The distance detection unit 507 detects the subject distance of the image data. That is, the distance detection unit 507 detects the subject distance from the imaging device 101 to the subject imaged as the combined ortho-cylindrical conversion image data. The imaging unit 501 includes a first fisheye lens 102a, a first imaging element 501a, a second fisheye lens 102b, and a second imaging element 501b. The angle of view of both the first fisheye lens 102a and the second fisheye lens 102b is 180 degrees or more.

[0073] Here, when the angle of view of both the first fisheye lens 102a and the second fisheye lens 102b is 200 degrees, in the range from 160 degrees to 200 degrees, the first image data generated by the output from the first imaging element 501a and the second image data generated by the output from the second imaging element 501b overlap.

[0074] The overlapping region is superimposed by the stitching in FIG. 7(E). By using triangulation for this overlapping region, the distance detection unit 507 can detect the subject distance in the overlapping region. For the angle of view outside the overlapping region, for example, a distance measurement sensor such as an ultrasonic sensor or LiDAR (Light Detection and Ranging) may be mounted as the sensor 404 to detect the subject distance outside the overlapping region.

[0075] In this case, the specifying unit 503 specifies a non-visible region based on a preset reference distance and the subject distance detected by the distance detection unit 507. The reference distance is the maximum distance to the inner peripheral wall inside the vehicle when the imaging device 101 is fixedly installed at an arbitrary position inside the vehicle, and is set in advance.

[0076] If the detected subject distance is equal to or greater than the reference distance, the specifying unit 503 determines that the subject exists outside the vehicle, and specifies the region where the subject image exists in the combined orthographic cylindrical conversion image data 107 as a visible region. Further, if the detected subject distance is less than the reference distance, the specifying unit 503 determines that the subject exists inside the vehicle, and specifies the region where the subject image exists in the combined orthographic cylindrical conversion image data 107 as a non-visible region.

[0077] FIG. 12 is an explanatory diagram showing a detection example of the subject distance in the overlapping region. In FIG. 12, the imaging device 101 calculates the subject distance D to a subject 1200 existing in an overlapping region where the angular fields of view of the first fisheye lens 102a and the second fisheye lens 102b overlap. Here, let the known interval on the optical axis OA of the first fisheye lens 102a and the second fisheye lens 102b be B, the intersection of the perpendicular line from the subject 1200 to the optical axis OA (the length of the perpendicular line is the subject distance D) and the optical axis OA be C, and the unknown distance on the optical axis OA from the installation position of the second fisheye lens 102b to the intersection C be X.

[0078] γ is the angular field of view of the first fisheye lens 102a with respect to the subject 1200, and α = 180 - γ / 2. β is the half angular field of view of the second fisheye lens 102b with respect to the subject 1200 corresponding to γ, and is uniquely determined when γ is determined. The subject distance D is expressed by the following formulas (1) and (2).

[0079] D = Qtanα ····· (1) D = (Q + B)tanβ ··· (2)

[0080] By eliminating the unknown distance Q from the above formulas (1) and (2), the subject distance D is obtained by the following formula (3).

[0081] D = B×(tanα×tanβ) / (tanα - tanβ) ··· (3)

[0082] Therefore, when the overlapping region is the visible region, the imaging device 101 can specify the subject distance to the subject outside the vehicle. For example, if the imaging device 101 is installed inside the vehicle so that the overlapping region becomes the visible regions of the front glass and the rear glass, the subject distances between the subjects in front of and behind the automobile 100 can be specified respectively.

[0083] Note that the distance detection unit 507 is specifically realized, for example, by causing the processor 401 to execute a program stored in the storage device 402, or by the LSI 405.

[0084] The calculation unit 508 calculates second information regarding the change in the posture of the imaging device 101 based on the non-visible region. Specifically, for example, the calculation unit 508 calculates the angular velocity from the motion vector mv of the non-visible region as the second information. Regarding this angular velocity as well, as described above, the amount of movement per unit time (x2 - x1, y2 - y1, z2 - z1) in the XYZ coordinate system obtained from the motion vector mv is calculated.

[0085] The coordinate values (x1, y1, z1) are the positions before movement, and the coordinate values (x2, y2, z2) are the positions before movement. The calculation unit 508 obtains the movement angle from the coordinate value (x1, y1, z1) to the coordinate value (x2, y2, z2) based on the coordinate values (x1, y1, z1), the coordinate values (x2, y2, z2), and the origin O, and differentiates it per unit time to calculate the angular velocity (ωy, ωp, ωr) as the second information as the movement angle per unit time. The angular velocity of the second information represents the movement inside the vehicle of the automobile 100. That is, since the angular velocity of the second information becomes the sway of the automobile 100 itself, it becomes a value much smaller than the angular velocity of the first information.

[0086] The calculation unit 508 is executed, for example, when the imaging device 101 is set with an automatic switching function between the first determination process and the second determination process. Therefore, when the angular velocity of the second information is calculated by the calculation unit 508, the determination unit 504 switches to and executes one of the determination processes based on the angular velocity of the first information and the angular velocity of the second information calculated by the calculation unit 508.

[0087] For example, if the angular velocity difference (which may also be a differentiated angle difference) is equal to or greater than a predetermined threshold value, the difference in the movement vectors mv inside and outside the vehicle will be large. Therefore, the determination unit 504 determines that the vehicle 100 is in motion. Thus, in order to suppress fluctuations in the extraction range TA as shown in (D) of FIGS. 1 to 3, if the current determination process is the second determination process, the determination unit 504 continues to execute the second determination process, and if the current determination process is the first determination process, it switches to and executes the second determination process. Thereby, the first determination process and the second determination process can be automatically switched, eliminating the hassle of manual switching by the user. Therefore, convenience can be improved.

[0088] Specifically, the calculation unit 508 is realized, for example, by causing the processor 401 to execute a program stored in the storage device 402 or by the LSI 405.

[0089] <Example of the processing procedure for manual mode switching> FIG. 13 is a flowchart showing an example of the processing procedure for manual mode switching by the imaging device 101. The imaging device 101 waits for a setting input by manual operation from the operation device 403 (step S1301: No). When there is a setting input (step S1301: Yes), the imaging device 101 determines whether the set mode is the sensor usage mode (step S1302).

[0090] When it is in the sensor usage mode (step S1302: Yes), the imaging device 101 switches the current mode to the sensor usage mode (step S1303). If the current mode is already the sensor usage mode, the imaging device 101 maintains the sensor usage mode. On the other hand, when it is not in the sensor usage mode (step S1302: No), the imaging device 101 switches the current mode to the image detection mode (step S1304). If the current mode is already the image detection mode, the imaging device 101 maintains the image detection mode. Thereby, the imaging device 101 finishes the manual mode switching.

[0091] In this way, by implementing the manual mode switching, for example, when the user holds and uses it, by setting it to the sensor usage mode through manual operation, using the posture of the imaging device 101 according to the detection signal from the sensor 404, as shown in FIGS. 1 to 3(D), the extraction position of the extraction range TA is changed to the extraction ranges 120, 130, 140 according to the inclination of the imaging device 101.

[0092] On the other hand, for example, when the imaging device 101 is mounted on the vehicle 100, by setting it to the image detection mode through manual operation, the imaging device 101 maintains the extraction position of the extraction range TA of the imaging device 101 as shown in FIGS. 1 to 3(C) at the extraction range 110 without using the posture of the imaging device 101 according to the detection signal from the sensor 404. That is, the imaging device 101 can ignore the influence of the acceleration, deceleration, and turning of the vehicle 100 and photograph as if the vehicle 100 had no acceleration, deceleration, or turning.

[0093] <Example of imaging processing procedure by the imaging device 101 in the sensor usage mode> FIG. 14 is a flowchart showing an example of an imaging processing procedure by the imaging device 101 in the sensor usage mode. The imaging device 101 waits for a start trigger (for example, input of an imaging start button) from the operation device 403 (step S1401: No). When there is a start trigger (step S1401: Yes), the acquisition unit 502 acquires an angular velocity based on a detection signal from the sensor 404, and obtains an angle by multiplying by a unit time (for example, the frame rate of the omnidirectional image data). Thereby, the acquisition unit 502 acquires a variation parameter for varying the position of the extraction range TA constituted by the direction indicated by the angular velocity and the angle (step S1402).

[0094] Also, the imaging device 101 acquires image data from the first imaging element 501a and the second imaging element 501b, and generates omnidirectional image data as shown in FIGS. 6 and 7 (step S1403). Then, the imaging device 101 associates the omnidirectional image data generated in step S1403 with the variation parameter acquired in step S1402 and stores it in the storage device 402 (step S1404).

[0095] The imaging device 101 waits for an end trigger (for example, input of an imaging end button) from the operation device 403 (step S1405). If there is no end trigger (step S1405: No), it returns to step S1402. By repeatedly executing steps S1402 to S1404, time-series omnidirectional image data, that is, video data can be stored in the storage device 402. When there is an end trigger (step S1405: Yes), the imaging device 101 ends the imaging of the omnidirectional image data.

[0096] By storing the omnidirectional image data and the variation parameter in association with each other, the extraction range TA at the time of reproduction varies in its position according to the variation direction (direction of the angular velocity) and the variation amount (angle) indicated by the variation parameter. Therefore, it becomes possible to display an image of the extraction range TA corresponding to the change in the posture of the imaging device 101.

[0097] <Example of the processing procedure for automatic mode switching> FIG. 15 is a flowchart showing an example of a processing procedure for automatic mode switching of the imaging device 101. In FIG. 15, the sensor usage mode is set as the default mode. The imaging device 101 waits for a start trigger (for example, an input of an imaging start button) from the operation device 403 (step S1501: No). When there is a start trigger (step S1501: Yes), as shown in FIG. 7(E), the combined orthographic cylindrical conversion image data Et is acquired (step S1502). The subscript t indicates the time-series number of the combined orthographic cylindrical conversion image data E at the time of acquisition in step S1502.

[0098] The imaging device 101 acquires a first angular velocity as first information regarding the attitude of the imaging device 101 from a detection signal from the sensor 404 by the acquisition unit 502 (step S1503). The imaging device 101 detects a motion vector mv using the acquired combined orthographic cylindrical conversion image data Et and the combined orthographic cylindrical conversion image data Et−1 acquired one time before in terms of time (step S1504). The imaging device 101 specifies a visible region and an invisible region in the combined orthographic cylindrical conversion image data Et from the detected motion vector mv by the specifying unit 503 (step S1505).

[0099] The imaging device 101 calculates a second angular velocity from the motion vector mv of the specified invisible region by the calculation unit 508 (step S1506). Then, the imaging device 101 obtains an angular velocity difference between the first angular velocity and the second angular velocity, and determines whether or not the absolute value of the angular velocity difference is greater than or equal to a threshold value (step S1507). Here, the angular velocity difference is used, but the first angular velocity and the second angular velocity may be converted into first angles and second angles by multiplying them by a unit time (for example, the frame rate of the omnidirectional image data), and the angle difference may be used.

[0100] If the absolute value of the angular velocity difference (or the angle difference) is less than the threshold value (step S1508: No), it is determined that the automobile 100 is not running, and the imaging device 101 switches to the sensor usage mode (step S1508). If the current mode is the sensor usage mode, the sensor usage mode is maintained.

[0101] On the other hand, if the absolute value of the angular velocity difference (or angular difference) is greater than or equal to the threshold value (step S1507: Yes), the vehicle 100 is in motion, and the imaging device 101 switches to the image detection mode (step S1509). If the current mode is the image detection mode, the image detection mode is maintained.

[0102] The imaging device 101 waits for an end trigger (for example, input of an imaging end button) from the operation device 403 (step S1510). If there is no end trigger (step S1510: No), the process returns to step S1502. By repeatedly executing steps S1502 to S1510, the behavior of the vehicle 100 can be constantly monitored, and automatic switching between the sensor usage mode and the image detection mode can be performed. If there is an end trigger (step S1510: Yes), the imaging device 101 ends the automatic mode switching.

[0103] In this way, in the automatic mode switching, automatic switching between the sensor usage mode and the image detection mode can be performed, so there is no trouble for the user to manually switch, and convenience can be improved.

[0104] <Example of Imaging Processing Procedure by Imaging Device 101 in Image Detection Mode> FIG. 16 is a flowchart showing an example of an imaging processing procedure 1 by the imaging device 101 in the image detection mode. The imaging processing procedure example 1 in FIG. 16 is an example of a processing procedure when the motion vector mv is detected by the motion detection unit 505. The imaging device 101 waits for a start trigger (for example, input of an imaging start button) from the operation device 403 (step S1601: No). If there is a start trigger (step S1601: Yes), as shown in (E) of FIG. 7, the combined orthographic cylindrical conversion image data Et is acquired (step S1502). The subscript t indicates the time-series number of the combined orthographic cylindrical conversion image data E at the time of acquisition in step S1602.

[0105] The imaging device 101 detects a motion vector mv using the combined orthographic cylindrical conversion image data Et acquired by the motion detection unit 505 and the combined orthographic cylindrical conversion image data Et-1 acquired one time before in terms of time (step S1603). The imaging device 101 specifies a visible region and an invisible region in the combined orthographic cylindrical conversion image data Et from the detected motion vector mv by the specifying unit 503 (step S1604).

[0106] The imaging device 101 calculates a second angular velocity from the motion vector mv of the invisible region in the combined orthographic cylindrical conversion image data Et by the calculating unit 508 (step S1605). The imaging device 101 calculates a second angle by multiplying the calculated second angular velocity by a unit time (for example, the frame rate of the omnidirectional image data) (step S1606). Since this second angle is an angle derived from the motion vector mv of the invisible region, it is a minute angle caused by vibrations of the automobile 100 or the like. This second angle becomes the variation amount of the variation parameter that varies the position of the extraction range TA.

[0107] Then, the imaging device 101 associates the omnidirectional image data Ft with the variation parameter (the direction and the second angle of the second angular velocity) and stores them in the storage device 402 (step S1404). The imaging device 101 waits for an end trigger (for example, an input of an imaging end button) from the operation device 403 (step S1608). If there is no end trigger (step S1608: No), the process returns to step S1602. By repeatedly executing steps S1602 to S1607, imaging in the image detection mode can be executed. If there is an end trigger (step S1608: Yes), the imaging device 101 ends the process in the image detection mode.

[0108] As described above, in the image detection mode, since the detection signal from the sensor 404 is not used, the imaging device 101 is not affected by the angular velocity or the angle derived from the detection signal from the sensor 404. Therefore, an image that does not move in position as shown in FIGS. 1 to 3(C) can be displayed without operating the electronic gimbal.

[0109] FIG. 17 is a flowchart showing an example of an imaging processing procedure 2 by the imaging device 101 in the image detection mode. The imaging processing procedure example 2 in FIG. 17 is an example of a processing procedure when the luminance is detected by the luminance detection unit 506. For the same processing as the imaging processing procedure example 1 in FIG. 16, the same step numbers are assigned and the description thereof is omitted.

[0110] The imaging device 101 detects the luminance from each region of the combined ortho-cylindrical conversion image data Et acquired in step S1602 by the luminance detection unit 506 (step S1703). The imaging device 101 specifies the visible region and the invisible region in the combined ortho-cylindrical conversion image data Et from the detected luminance by the specifying unit 503 (step S1704).

[0111] The imaging device 101 obtains the motion vector mv from the combined ortho-cylindrical conversion image data Et, Et-1 by the calculation unit 508, and calculates the second angular velocity from the motion vector mv of the invisible region in the combined ortho-cylindrical conversion image data Et (step S1705).

[0112] As described above, in the image detection mode, since the detection signal from the sensor 404 is not used, the imaging device 101 is not affected by the angular velocity or angle derived from the detection signal from the sensor 404. Therefore, an image that does not move as shown in FIGS. 1 to 3(C) can be displayed without operating the electronic gimbal.

[0113] In general, the luminance inside the vehicle is lower than that outside the vehicle. For example, during the day, sunlight is blocked by the ceiling of the interior space, so the luminance inside the vehicle is lower than that outside the vehicle. Also, at night, the inside of the vehicle is brighter than the outside due to the headlight of the automobile 100 and the outside lighting, so the luminance inside the vehicle is lower than that outside the vehicle. By utilizing such properties, it is possible to specify the visible region and the invisible region only with the combined ortho-cylindrical conversion image data Et without using the motion vector mv detection, and the processing speed can be improved.

[0114] FIG. 18 is a flowchart showing an example of an imaging processing procedure 3 by the imaging device 101 in the image detection mode. The imaging processing procedure 3 in FIG. 17 is an example of a processing procedure when the luminance is detected by the distance detection unit 507. Note that the same processing as that of the imaging processing procedure 1 in FIG. 16 is given the same step number, and the description thereof is omitted.

[0115] The imaging device 101 detects the subject distance from the overlapping area at the time of stitching in the combined ortho-cylindrical conversion image data Et by the distance detection unit 507 (step S1803). The imaging device 101 compares the preset reference distance with the subject distance, and specifies a visible area and an invisible area within the overlapping area (step S1804).

[0116] The imaging device 101 obtains a motion vector mv from each overlapping area of the combined ortho-cylindrical conversion image data Et and Et-1 by the calculation unit 508, and calculates a second angular velocity from the motion vector mv of the invisible area within the overlapping area in the combined ortho-cylindrical conversion image data Et (step S1805).

[0117] As described above, in the image detection mode, since the detection signal from the sensor 404 is not used, the imaging device 101 is not affected by the angular velocity or angle derived from the detection signal from the sensor 404. Therefore, an image that does not move as shown in FIGS. 1 to 3(C) can be displayed without operating the electronic gimbal.

[0118] In addition, in order to obtain the subject distance of the overlapping area, it is possible to spatially specify whether the subject exists inside or outside the vehicle. Further, since the range of the combined ortho-cylindrical conversion image data Et to be processed to obtain the variation parameter is limited to the overlapping area, it is not necessary to use the entire range of the combined ortho-cylindrical conversion image data Et, and the efficiency of the processing in the image detection mode can be improved.

[0119] <Example of a playback processing procedure for omnidirectional image data> FIG. 19 is a flowchart showing an example of a reproduction processing procedure for omnidirectional image data. The reproduction processing of the omnidirectional image data is executed, for example, on a terminal 400 which is the transfer destination of the omnidirectional image data and the variation parameters. The terminal 400 waits for a reproduction start trigger (step S1901: No). When there is a reproduction start trigger (step S1901: Yes), the terminal 400 reads the omnidirectional image data Fi at the i-th reproduction position and the associated variation parameters (step S1902).

[0120] The terminal 400 updates the extraction range TA based on the omnidirectional image data Fi and the associated variation parameters (step S1903). The terminal 400 displays the output target image data 710 within the updated extraction range TA after step S1903 on the display screen of the terminal 400. The terminal 400 increments the reproduction position i (step S1905) and determines whether an end trigger has been received (step S1906).

[0121] If the end trigger has not been received (step S1906: No), the process returns to step S1902. Thereby, steps S1902 to S1904 are executed for the next omnidirectional image data Fi. If the end trigger has been received (step S1906: Yes), the terminal 400 ends the reproduction process.

[0122] In this way, since the omnidirectional image data and the variation parameters are associated with each other, it is possible to display the output target image data 710 while updating the extraction range TA during reproduction. Therefore, for the omnidirectional image data captured in the sensor usage mode, the position of the extraction range TA varies to the extraction ranges 120, 130, 140 as shown in FIGS. 1 to 3 (D), and the output target image data 710 is displayed. For the omnidirectional image data captured in the image detection mode, the position of the extraction range TA is maintained at the extraction range 110 as shown in FIGS. 1 to 3 (C), and the output target image data 710 is displayed.

[0123] <Application Example to a Motorcycle> FIG. 20 is an explanatory diagram showing an imaging example by the imaging device 101 mounted on a motorcycle. (A) is a side view of the motorcycle 2000 equipped with the imaging device 101, (B) is a plan view of the motorcycle 2000 equipped with the imaging device 101, and (C) is a front view of the motorcycle 2000 equipped with the imaging device 101. (C) shows a state in which the motorcycle 2000 is tilted due to the centrifugal force c of turning.

[0124] As shown in (A), the gravitational acceleration g acts on the imaging device 101. As shown in (B), when turning left, the centrifugal force c of turning acts on the imaging device 101. When turning, since the motorcycle 2000 tilts, as shown in (C), the resultant acceleration gc of the gravitational acceleration g and the centrifugal force c of turning is obtained.

[0125] (D) shows an imaging example of a subject by the imaging device 101 when the centrifugal force gc due to turning is not applied. Specifically, for example, (D) shows the extraction range 2001 of the omnidirectional image data and the output target image data 2010 extracted from the omnidirectional image data within the extraction range 2001, when the imaging direction CD1 is set as the projection direction of the stitched omnidirectional image data. The output target image data 2010 includes the image data 2011 of the motorcycle 2000 without movement and the image data 2012 of the landscape with movement.

[0126] (E) shows an imaging example of a subject by the imaging device 101 when the centrifugal force gc due to turning is applied. Specifically, for example, (E) shows the extraction range 2002 of the omnidirectional image data and the output target image data 2020 extracted from the omnidirectional image data within the extraction range 2002, when the imaging direction CD1 is set as the projection direction of the stitched omnidirectional image data. The output target image data 2020 includes the image data 2021 of the motorcycle 2000 without movement and the image data 2022 of the landscape with movement.

[0127] In (D), when the imaging device 101 is in the sensor usage mode, if the motorcycle 2000 tilts as in (C), the electronic gimbal of the imaging device 101 corrects the tilt according to the combined acceleration gc, and thus generates a variation parameter for varying the position of the extraction range TA. That is, although the motorcycle 2000 is tilted, the electronic gimbal corrects the position of the extraction range TA as if it were not tilted. Therefore, due to the correction by the electronic gimbal, the imaging device 101 has the extraction range 2001 shown in (D).

[0128] In (D), when the imaging device 101 is in the image detection mode, if the motorcycle 2000 tilts as in (C), the electronic gimbal of the imaging device 101 does not correct the tilt according to the combined acceleration gc, and thus generates a variation parameter for hardly varying the position of the extraction range TA. Therefore, since the correction by the electronic gimbal is not applied, the imaging device 101 has the extraction range 2002 shown in (E). In this way, when applied to the motorcycle 2000, output target image data 710 that takes into account the tilt of the motorcycle 2000 due to turning can be obtained, enabling the display of a dynamic touring video.

[0129] (1) As described above, according to the imaging device 101 according to this embodiment, a first determination process (for example, a process in the sensor usage mode) for determining a variation parameter for varying the position of the extraction range TA of the output target image data 710 from the image data based on first information (for example, a first angular velocity) regarding a change in the posture of the imaging device 101, and a second determination process different from the first determination process can be automatically switched.

[0130] (2) Further, based on the non-visible region, second information (for example, a second angular velocity) regarding a change in the posture of the imaging device 101 is calculated, and by executing either one of the determination processes, automatic switching between the first determination process and the second determination process can be realized according to the behavior of the moving body.

[0131] (3) Also, by making the second determination process a process of stopping the output of the data detected by the sensor 404, the data from the sensor 404 can be forcibly blocked and the second determination process can be executed.

[0132] (4) Also, by determining a variation parameter for varying the position of the extraction range TA based on the image data, the second determination process can be executed without using the data from the sensor 404.

[0133] (5) Also, by determining a variation parameter for varying the position of the extraction range TA of the output target image data 710 from the image data without using the first information (for example, the first angular velocity) regarding the change in the posture of the imaging device 101, the second determination process can be executed, for example, as an image detection mode.

[0134] (6) Also, without using the data obtained from the sensor 404, the motion vector mv and the invisible region can be specified, and the variation in the position of the extraction range due to the data obtained from the sensor 404 can be suppressed.

[0135] (7) Also, the invisible region can be specified without detecting the motion vector mv, and the speed of the specifying process can be increased.

[0136] (8) Also, based on the reference distance to the inner peripheral wall in the moving body and the subject distance, by specifying the invisible region, the invisible region can be specified without detecting the motion vector mv, and the speed of the specifying process can be increased.

[0137] (9) Further, by calculating the second information regarding the change in the posture of the imaging device 101 and determining a variation parameter for varying the position of the extraction range TA, it is possible to identify, via the invisible region, the behavior within the moving body (for example, vibration) rather than the behavior of the moving body itself (for example, acceleration, deceleration, or turning), and reflect it in the extraction range TA. Therefore, when there is no vibration, there is no variation in the extraction range TA, and when there is vibration, the extraction range TA will vary following the vibration.

[0138] (10) Also, by identifying the invisible region based on the direction and magnitude of the motion vector mv, it is possible to improve the identification accuracy of the invisible region.

[0139] (11) Also, by detecting the motion vector mv based on the moving angle of the region of interest from the combined orthographic cylindrical conversion image data Et-1 to the combined orthographic cylindrical conversion image data Et, since the calculation is in the polar coordinate system, it is possible to improve the detection accuracy of the motion vector mv.

[0140] (12) Also, by detecting the subject distance D based on the overlapping range between the first orthographic cylindrical conversion image data 602a and the second orthographic cylindrical conversion image data 602b, it is possible to spatially identify whether the subject exists inside or outside the moving body. Also, since the range of processing to be performed for obtaining the variation parameter is limited to the overlapping region, it is not necessary to use the entire range of the combined orthographic cylindrical conversion image data, and the efficiency of the second determination process can be improved.

[0141] (13) Further, according to the control program according to this embodiment, the first determination process for determining a variation parameter for varying the position of the extraction range TA of the output target image data 710 from the image data based on the information (for example, angular velocity) regarding the change in the posture of the imaging device 101, and the second determination process different from the first determination process can be automatically switched by software. Therefore, even for an imaging device in which the control program is not installed, by installing the control program, the imaging device can execute the automatic switching between the first determination process and the second determination process.

[0142] Also, the image data (e.g., omnidirectional image data) captured by the imaging device 101 described above is associated with the variation parameter. When the playback device (e.g., the terminal 400) designates the extraction range TA from the omnidirectional image data and plays back the output target image data, the position of the extraction range TA is controlled by the variation parameter. That is, if it is the variation parameter obtained by the first determination process, the position of the extraction range TA varies, but if it is the variation parameter obtained by the second determination process, the position of the extraction range TA hardly varies.

[0143] Thus, when the imaging device 101 is not mounted on a moving body, the first determination process is executed. On the other hand, when the imaging device 101 is mounted on a moving body, since it automatically switches from the first determination process to the second determination process and executes it, by stopping the overcorrection of the electronic gimbal, it is possible to display omnidirectional image data with a stable imaging direction.

Explanation of Signs

[0144] 100 Automobile, 101 Imaging device, 102a First fisheye lens, 102b Second fisheye lens, 107,701 Combined positive-distance cylindrical conversion image data, TA,110,120,130,140,2001,2002 Extraction range, 111,121,131,141,710,2010,2020 Output target image data, 400 Terminal, 401 Processor, 402 Storage device, 403 Operation device, 404 Sensor, 406 Omnidirectional camera, 501 Imaging unit, 501a First imaging element, 501b Second imaging element, 501c Image processing unit, 502 Acquisition unit, 503 Identification unit, 504 Determination unit, 505 Detection unit, 506 Luminance detection unit, 507 Distance detection unit, 508 Calculation unit, 1200 Subject, 2000 Motorcycle

Claims

1. An imaging device mountable on a moving body, comprising: an imaging unit that images a subject; an acquisition unit that acquires, by a sensor, first information regarding a change in the attitude of the imaging device; a specifying unit that specifies a non-visible region where the outside of the moving body cannot be visually recognized, based on the image data of the subject generated by the output from the imaging unit; a first determination process that determines a variation parameter for varying the position of an extraction range of output target image data from the image data, based on the first information acquired by the acquisition unit, and a second determination process different from the first determination process, wherein, based on second information indicating a change in the attitude of the imaging device obtained from a motion vector in the non-visible region specified by the specifying unit and the first information, a determination unit executes either one of the determination processes; The second determination process is a process of invalidating a detection signal of the sensor or a process of determining a variation parameter for varying the position of the extraction range based on the image data. The imaging device.

2. In the imaging device according to claim 1, the determination unit executes either one of the determination processes based on a difference between a value of the first information and a value of the second information. The imaging device.

3. The imaging device according to claim 1, wherein the process of invalidating the detection signal of the sensor is a process of stopping the output of data detected by the sensor. The imaging device.

4. The imaging device according to claim 1, wherein the process of invalidating the detection signal of the sensor is to determine a variation parameter for varying the position of an extraction range of output target image data from the image data without using the first information. The imaging device.

5. The imaging device according to claim 1, comprising a motion detection unit that detects the motion vector based on a time-series image data group generated by the output from the imaging unit; wherein the specifying unit specifies the non-visible region based on the motion vector detected by the motion detection unit. The imaging device.

6. The imaging device according to claim 1, comprising a luminance detection unit that detects the luminance of the image data; wherein the specifying unit specifies the non-visible region based on the luminance detected by the luminance detection unit. The imaging device.

7. The imaging device according to claim 1, comprising a distance detection unit that detects the subject distance of the image data. The imaging device, wherein the specific part specifies the invisible region based on a reference distance to an inner peripheral wall within the moving body and a subject distance detected by the distance detection part.

8. The imaging device according to claim 5, wherein the determination part determines a variation parameter for varying the position of the extraction range using the second information in the second determination process.

9. The imaging device according to claim 5, wherein the specific part specifies the invisible region based on the direction and magnitude of the motion vector.

10. The imaging device according to claim 5, wherein the motion detection part detects the motion vector based on a movement angle of a region of interest from first image data in the time-series image data group to second image data that is temporally later than the first image data.

11. The imaging device according to claim 7, wherein the imaging part includes a first fisheye lens, a first imaging element that images a first subject via the first fisheye lens, a second fisheye lens arranged such that the first fisheye lens and a concave surface face each other, and a second imaging element that images a second subject via the second fisheye lens, and the distance detection part detects the subject distance based on an overlapping range between first image data generated from an output of the first imaging element and second image data generated from an output of the second imaging element.

12. In the imaging device according to claim 1, the process of invalidating the detection signal of the sensor is a process of not using the detection signal from the sensor.

13. In the imaging device according to claim 12, the process of not using the detection signal from the sensor is a process of stopping the output of the detection signal from the sensor or a process of receiving the detection signal from the sensor but not using the detection signal.

14. A control program for causing a processor to execute control of an imaging device that can be mounted on a moving body and images a subject, wherein the processor is caused to acquire first information regarding a change in the posture of the imaging device by a sensor, specify an invisible region where the outside of the moving body cannot be visually recognized based on the image data of the subject imaged by the imaging device. A first determination process for determining a variation parameter for varying the position of an extraction range of output target image data from the image data based on the first information, and among the first determination process and a second determination process different from the first determination process, based on second information indicating a change in the posture of the imaging device obtained from a motion vector in the invisible region and the first information, cause either one of the determination processes to be executed, The second determination process is a control program that is a process for invalidating a detection signal of the sensor or a process for determining a variation parameter for varying the position of the extraction range based on the image data.

Images