Imaging device and its control method

Abstract

To provide an image pickup apparatus, a portable device, a calibrator, a control method therefor, and a program, capable of eliminating manual change of an image pickup direction during picking up an image and capable of easily obtaining an image that records experience even while focusing attention on the experience.SOLUTION: A camera body 1 includes: a face direction detection unit 20 that is worn on a body other than a head of a user and detects an observation direction of the user; an image-capturing unit 40 that is worn on the body of the user and picks up an image; and a transmission unit 70 that outputs an image corresponding to the observation direction among the images picked up by the image-capturing unit 40.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention is an imaging device and Control method In the law Regarding this, in particular, imaging devices used as action cameras and Control method In the law To relate to. [Background technology]

[0002] Traditionally, when taking images with a camera, the photographer needs to keep the camera pointed towards the subject being photographed. This means that the photographer's hands are occupied with the act of taking the image, preventing them from doing anything else, or concentrating their attention on taking the image prevents them from fully experiencing the moment.

[0003] For example, in terms of image capture operations, a parent who is taking the picture cannot play with their child while capturing images of the child, and conversely, if they try to play with their child, they cannot capture images, which presents a challenge.

[0004] Furthermore, in terms of focusing attention on imaging, when imaging during a sporting event, the photographer may not be able to cheer or remember the details of the game, and focusing attention on watching the sport may prevent them from imaging. Similarly, when imaging during a group trip, the photographer may not be able to experience the emotions at the same level as the other members, and prioritizing the experience may lead to neglecting imaging.

[0005] Therefore, Patent Document 1 discloses a technique that uses a second camera to image the user in addition to a first camera to image the subject. In this technique, the user's direction of movement and gaze direction are calculated from the image captured by the second camera, the imaging direction of the first camera is determined, and the subject is estimated and imaged based on the user's preferences and state.

[0006] Furthermore, Patent Document 2 discloses an image recording system in which a sensor consisting of a gyroscope and an accelerometer is attached to the head to detect the observation direction of the photographer (user), and an imaging device attached separately to the body or a bag captures images of the observation direction detected by the sensor. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2007-74033 [Patent Document 2] Japanese Patent Publication No. 2017-60078 [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] However, in Patent Document 1, since the second camera captures images of the user from a position away from the user, high optical performance is required for the second camera to calculate the user's direction of movement and gaze direction from the images captured by the second camera. Furthermore, image processing of the images captured by the second camera requires high computational processing power, resulting in a large and complex device. Moreover, even with these measures, it is not possible to precisely calculate the user's observation direction, making it impossible to accurately estimate the subject based on the user's preferences and state, resulting in the capture of images that do not match the image the user desires.

[0009] Furthermore, in Patent Document 2, since the user's observation direction is detected directly, the user needs to wear a sensor on their head, even if it's just a sensor, which does not eliminate the inconvenience of wearing any device on the head as described above. Also, when the sensor consists of a gyroscope or accelerometer, a certain level of accuracy can be achieved in detecting the relative observation direction, but accuracy in detecting the absolute observation direction, especially the horizontal rotation direction, cannot be achieved, so there were challenges to practical application.

[0010] Therefore, the objective of the present invention is to provide an imaging device that eliminates the need to manually change the imaging direction during imaging, allowing users to concentrate on the experience while easily acquiring video footage of that experience. and Control method Law It is about providing. [Means for solving the problem]

[0011] The imaging device according to claim 1 of the present invention comprises: an observation direction detection means attached to the user's body other than the head for detecting the user's observation direction; an imaging means attached to the user's body for capturing an image; and an image output means that outputs an image corresponding to the observation direction based on the image captured by the imaging means. The observation direction detection means comprises an infrared irradiation means that irradiates an infrared irradiation surface including the user's chin with infrared light, and an infrared detection means that detects the reflected light of the infrared light reflected from the infrared irradiation surface. The infrared detection means detects the position of the user's neck rotation center and chin from the reflected light of the infrared light detected, and detects the observation direction from the positions of the neck rotation center and chin. It is characterized by the following: [Effects of the Invention]

[0014] According to the present invention, it is unnecessary to manually change the imaging direction during imaging, and it is possible to easily acquire video footage of the experience while concentrating on the experience itself. [Brief explanation of the drawing]

[0015] [Figure 1A] This is an external view of the camera body, including the imaging and detection unit, as an imaging device according to Example 1. [Figure 1B] This diagram shows the camera body being worn by the user. [Figure 1C] This is a view of the battery section of the camera body from the rear, as shown in Figure 1A. [Figure 1D] This is an external view of a display device as a portable device according to Embodiment 1, which is composed of a separate component from the camera body. [Figure 2A] This is a front view of the shooting and detection unit in the camera body. [Figure 2B] This diagram shows the shape of the band portion of the connection part on the camera body. [Figure 2C] This is a view of the imaging and detection unit from the back. [Figure 2D] This is a top view of the imaging and detection unit. [Figure 2E] This diagram shows the configuration of the face direction detection unit, which is located inside the shooting and detection unit and positioned below the face direction detection window in the camera body. [Figure 2F] This is a view from the left side of the user, showing the camera body as it is worn by the user. [Figure 3] This is a diagram illustrating the details of the battery section. [Figure 4] This is a functional block diagram of the camera body according to Example 1. [Figure 5] This is a block diagram showing the hardware configuration of the camera body. [Figure 6] Block diagram showing the hardware configuration of a display device. [Figure 7A] This flowchart shows an overview of the image recording process according to Example 1, which is performed in the camera body and display device. [Figure 7B] This is a flowchart of the subroutine for the preparation operation process in step S100 of Figure 7A, according to Example 1. [Figure 7C] This is a flowchart of the subroutine for the face direction detection process in step S200 of Figure 7A, according to Example 1. [Figure 7D] This is a flowchart of the subroutine for determining the recording direction and range in step S300 of Figure 7A, according to Example 1. [Figure 7E] This is a flowchart of the subroutine for the recording range development process in step S500 of Figure 7A, according to Example 1. [Figure 7F] This diagram illustrates the process from steps S200 to S600 in Figure 7A in video mode. [Figure 8A] This diagram shows the user's image as seen through the face direction detection window. [Figure 8B] This diagram shows the case where a fluorescent light in the room is reflected as a background in the image of the user as seen through the face direction detection window. [Figure 8C]Figure 8B shows the image obtained when the user and the fluorescent lamp in the background shown in Figure 8B are imaged by the infrared detection device's sensor through the face direction detection window, with the infrared LED of the infrared detection device not lit. [Figure 8D] Figure 8B shows the image obtained when the user and the fluorescent lamp in the background shown in Figure 8B are imaged by the sensor of the infrared detection processing device through the face direction detection window, with the infrared LED turned on. [Figure 8E] This figure shows the difference image calculated by subtracting the image in Figure 8C from the image in Figure 8D. [Figure 8F] This figure shows the result of adjusting the contrast of the difference image in Figure 8E to match the light intensity of the infrared reflected light projected onto the user's face and neck. [Figure 8G] Figure 8F shows a diagram in which symbols indicating different parts of the user's body, along with symbols for a double circle indicating the neck position and a black circle indicating the chin position, superimposed on each other. [Figure 8H] This figure shows the difference image calculated using the same method as in Figure 8E, when the user's face is turned to the right. [Figure 8I] The contrast of the difference image in Figure 8H has been adjusted to match the light intensity of the infrared reflected light projected onto the user's face and neck, and the double circle indicating the neck position and the black circle indicating the chin position are superimposed on the image. [Figure 8J] This diagram shows the user's image as seen through the face direction detection window when the user's face is turned 33° upward from the horizontal. [Figure 8K] This diagram shows the difference in image density calculated in the same way as in Figure 8E, when the user's face is turned 33° above the horizontal, adjusted to the light intensity of the infrared reflected light projected onto the user's face and neck, and superimposed with a double circle indicating the neck position and a black circle indicating the chin position. [Figure 9] This is a timing chart showing the timing of infrared LED illumination and related signals. [Figure 10] This diagram illustrates the vertical movement of the user's face. [Figure 11A]This diagram shows the target field of view in an ultra-wide-angle image captured by the camera's shooting unit when the user is facing forward. [Figure 11B] This figure shows the image of the target field of view extracted from the ultra-wide-angle image in Figure 11A. [Figure 11C] This diagram shows the target field of view in an ultra-wide-angle image when the user is observing subject A. [Figure 11D] This figure shows the image of the target field of view, extracted from ultra-wide-angle footage in Figure 11C, with distortion and shaking corrected. [Figure 11E] This figure shows the target field of view in ultra-wide-angle video when the user is observing subject A with a field of view setting smaller than that shown in Figure 11C. [Figure 11F] This figure shows the image of the target field of view, extracted from ultra-wide-angle footage in Figure 11E, with distortion and shaking corrected. [Figure 12A] This figure shows an example of the target field of view in ultra-wide-angle video. [Figure 12B] This figure shows an example of a target field of view in ultra-wide-angle video, where the field of view setting is the same as the target field of view in Figure 12A, but the observation direction is different. [Figure 12C] This figure shows another example of a target field of view in ultra-wide-angle video, with the same field of view setting as the target field of view in Figure 12A, but with a different observation direction. [Figure 12D] This figure shows an example of a target field of view in ultra-wide-angle video, where the observation direction is the same as the target field of view in Figure 12C, but the field of view setting value is smaller. [Figure 12E] This figure shows an example where a pre-vibration isolation area corresponding to a predetermined vibration isolation level is added around the target field of view shown in Figure 12A. [Figure 12F] This figure shows an example where a pre-vibration isolation area corresponding to the same vibration isolation level as the pre-vibration isolation area in Figure 12E is added around the target field of view shown in Figure 12B. [Figure 12G] This figure shows an example where a pre-vibration isolation area corresponding to the same vibration isolation level as the pre-vibration isolation area in Figure 12E is added around the target field of view shown in Figure 12D. [Figure 13]This diagram shows the menu screen for various video mode settings, which is displayed on the display unit of the camera before image capture. [Figure 14] Figure 7A is a flowchart of the subroutine for the primary recording process in step S600. [Figure 15] This diagram shows the data structure of the video file generated by the primary recording process. [Figure 16] Figure 7A is a flowchart of the subroutine for the transfer process to the display device in step S700. [Figure 17] Figure 7A is a flowchart of the subroutine for the optical correction process in step S800. [Figure 18] This figure illustrates the process of performing distortion correction in step S803 of Figure 17. [Figure 19] Figure 7A is a flowchart of the vibration isolation subroutine in step S900. [Figure 20] This figure shows the details of the calibrator used in the calibration process according to Example 2. [Figure 21] This is a flowchart of the calibration process according to Embodiment 2, which is performed in the camera body and calibrator. [Figure 22A] This figure shows the screen displayed on the calibrator's display unit during the calibration operation in the direction facing the user, in step S3103 of Figure 21. [Figure 22B] This figure shows the user holding the calibrator forward in accordance with the instructions shown in the instruction display in Figure 22A. [Figure 22C] This is a schematic diagram showing the entire ultra-wide-angle image captured by the imaging lens in the state shown in Figure 22B. [Figure 22D] Figure 22C is a schematic diagram showing an image with aberrations corrected from the ultra-wide-angle image shown. [Figure 22E] This is a schematic diagram showing the face direction image acquired by the face direction detection unit in step S3108 of Figure 21 during the calibration operation for the user's frontal direction. [Figure 22F] This is a schematic diagram showing the in-camera image displayed in step S3107 of Figure 21. [Figure 23A] This figure shows the screen displayed on the calibrator's display unit during the user's upward right-hand calibration operation in step S3103 of Figure 21. [Figure 23B] This figure shows the user holding the calibrator to the upper right in accordance with the instructions shown in the instruction display in Figure 23A. [Figure 23C] This is a schematic diagram showing the entire ultra-wide-angle image captured by the imaging lens in the state shown in Figure 23B. [Figure 23D] Figure 23C is a schematic diagram showing an image with aberrations corrected from the ultra-wide-angle image shown. [Figure 23E] This is a schematic diagram showing the face direction image acquired by the face direction detection unit in step S3108 of Figure 21 during the calibration operation of the user's right hand in the upward direction. [Figure 24] This is a diagram illustrating the delayed image extraction in Example 3. [Figure 25] This figure shows the trajectory of facial movement that is maintained in Example 3. [Figure 26] This is a flowchart of the motion sickness prevention process according to Example 3. [Figure 27] This graph illustrates the cropping range correction process according to Example 4. [Figure 28] (a) is a flowchart showing the recording direction and range determination process according to Example 4, and (b) is a flowchart showing the cropping range correction process in step S400 of (a). [Figure 29] This is a schematic diagram illustrating the relationship between the user's field of view and the target field of view when observing a close-range subject in Example 1. [Figure 30] This is an external view of the camera body including the imaging device according to Example 5. [Figure 31] This is a block diagram showing the hardware configuration of the camera body according to Example 5. [Figure 32]This is a schematic diagram illustrating the relationship between the user, calibrator, and target field of view during calibration, including parallax correction mode processing, in Example 5. [Figure 33A] This is a flowchart of the parallax correction mode processing, which is part of the preparation process in step S100 of Figure 7A in Example 5. [Figure 33B] This is a flowchart of the recording direction and range determination subroutine for S300, as explained in Figure 7A of Example 5. [Figure 34] This is a schematic diagram showing the relationship between the defocus map created in step S5302 of Figure 33B and the recording direction. [Figure 35] This is a flowchart of the observation direction determination process according to Example 6. [Figure 36A] This figure shows the relationship between the user's observation direction detection state for each frame and the captured image, according to Example 6. [Figure 36B] This figure shows the relationship between the user's observation direction detection state for each frame and the captured image in the subject loss mode according to Example 6. [Figure 37] This is a diagram illustrating the relationship between the observation direction and the face region that can be used to detect the face direction, according to Example 7. [Figure 38] This is a flowchart of the observation direction determination process when acquiring face direction according to Example 7, which is performed instead of the process in step S6004 in Figure 35. [Figure 39] This figure shows the relationship between face direction and face direction reliability in Example 7. [Figure 40] This is a conceptual diagram of the observation direction determination process when acquiring face direction in Example 7. [Figure 41] This is an enlarged view showing the imaging and detection unit from the side. [Figure 42] This is a side view showing the camera body attached to the user. [Figure 43] This is an enlarged view showing the shooting / detection unit from the side, with the connection part hidden. [Figure 44]This is a side view showing how the camera body looks when the connection part is hidden and the user has attached it. [Figure 45] This figure shows the band portion and the connection surface, which is the cut surface of the electrical cable that is integrally formed with it. [Figure 46A] This is a block diagram showing the hardware configuration of a display device connected to a camera body, including an imaging device according to Example 9. [Figure 46B] This is a functional block diagram of the camera body according to Example 9. [Figure 47] This is a functional block diagram of the camera body and display device according to Example 10. [Figure 48] This flowchart shows an overview of the image recording process according to Example 10, which is performed in the camera body and display device. [Figure 49] This is a functional block diagram of the camera body and display device according to Example 11. [Figure 50] This flowchart shows an overview of the image recording process according to Example 11, which is performed in the camera body and display device. [Figure 51A] This is an external view of the camera body in Example 12. [Figure 51B] This is a perspective view showing the details of the shooting and detection unit, which is part of the camera body in Example 12. [Figure 51C] Figure 51B is a perspective view showing the imaging unit of the imaging and detection unit rotated 30 degrees to the left. [Figure 51D] Figure 51C is a perspective view showing the imaging unit facing downwards at a 30-degree angle. [Figure 52] This is a functional block diagram of the camera body according to Example 12. [Figure 53] This is a block diagram showing the hardware configuration of the camera body according to Example 12. [Figure 54] This flowchart shows an overview of the image recording process according to Example 12, which is performed in the camera body and display device. [Figure 55] This is a flowchart of the subroutine for the imaging unit drive process in step S12300 of Figure 54, relating to Example 12. [Figure 56] This is a flowchart of the development process subroutine in step S12500 of Figure 54, relating to Example 12. [Figure 57] This is a block diagram showing the hardware configuration of the camera body according to Example 13. [Figure 58] This is a schematic diagram illustrating an example of a training image used in Example 13. [Figure 59] This is a flowchart illustrating the process of detecting face direction using machine learning according to Example 13. [Figure 60] This is a block diagram showing the hardware configuration of the camera body according to Example 14. [Figure 61] (a) is a schematic diagram showing a distance image generated when the ToF device of the camera body according to Example 14 is installed at the clavicle position and the ToF device measures and captures images in the upward direction; (b) is a schematic diagram showing an image obtained by applying threshold processing to the distance image in (a) to extract the face portion; (c) is a schematic diagram showing an image obtained by dividing the image in (b) into regions according to the distance information; and (d) is a schematic diagram showing an image obtained from the image in (c) that shows the neck position and chin position. [Figure 62] This is a flowchart illustrating the calculation process for face direction in Example 14. [Figure 63] This figure shows an example of a camera configuration that is fixed to the head using a conventional head-fixing accessory. [Figure 64] This figure shows an example configuration of a conventional 360-degree camera. [Figure 65] This figure shows an example of the conversion process for images captured by a 360-degree camera (Figure 48). [Modes for carrying out the invention]

[0016] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0017] First, let's consider a few ways to address two challenges: recording important moments and allowing the photographer to focus on the experience. One method is to use a head-mounting accessory to fix the action camera to the head and capture images in the direction the photographer is observing, thus allowing the photographer to capture images without being distracted by the camera's hands. Another method is to use a 360-degree camera to capture a wide area, allowing the photographer to focus on the experience while it is happening, and then, after the experience is over, to cut out and edit the necessary footage from the captured 360-degree video to preserve the video of the experience.

[0018] The former method requires the cumbersome act of attaching a head-mounting accessory 902, to which the action camera 901 body is fixed, to the head, as shown in Figure 63(a). Furthermore, as shown in Figure 63(b), when the photographer attaches the action camera 901 to their head using the head-mounting accessory 902, it looks unsightly and causes problems such as messing up the photographer's hairstyle. In addition, the photographer was bothered by the weight and presence of the head-mounting accessory 902 and the action camera 901 attached to their head, and was also concerned about the unsightly appearance to third parties. As a result, in the state shown in Figure 63(b), the photographer was unable to concentrate on the experience, or felt resistance to being in the state shown in Figure 63(b), making it difficult to take images.

[0019] On the other hand, the latter method requires a series of operations such as image conversion and specifying the cropping position. For example, a 360-degree camera 903 equipped with a lens 904 and a shooting button 905 is known, as shown in Figure 64. The lens 904 is one of a pair of hemispherical fisheye lenses configured on both sides of the housing of the 360-degree camera 903, and the 360-degree camera 903 performs 360-degree photography using this pair of fisheye lenses. Then, a 360-degree image is obtained by combining the images captured using this pair of fisheye lenses.

[0020] Figure 65 shows an example of the conversion process for images captured by the 360-degree camera 903.

[0021] Figure 65(a) shows an example of a 360-degree image obtained by the 360-degree camera 903, which includes the subjects: the photographer 906, the child 907, and the tree 908. Because this image is a 360-degree image obtained by combining the projection images of a pair of fisheye lenses, the photographer 906 is greatly distorted. Also, the child 907, the subject that photographer 906 was trying to photograph, has its body greatly distorted and stretched from side to side because its body was located at the periphery of the shooting range of lens 904. On the other hand, the tree 908 was positioned directly in front of lens 904, so it was photographed without much distortion.

[0022] To create an image representing the field of view that a person normally sees from the image in Figure 65(a), it is necessary to cut out a portion of it, transform it into a plane, and display it.

[0023] Figure 65(b) is an image extracted from the image in Figure 65(a) that is positioned directly in front of the lens 904. In the image in Figure 65(b), the tree 908 is in the center, representing a field of view similar to that of a normal person. However, the child 907 that the photographer 906 was trying to capture is not included in the image in Figure 65(b), so the extraction position must be changed. Specifically, the extraction position in Figure 65(a) must be to the left of the tree 908 and 30° downward from the perspective of the drawing. After this extraction work is performed, the image is transformed into a plane and displayed as Figure 65(c). Thus, in order to obtain the image in Figure 65(c) that the photographer was trying to capture from the image in Figure 65(a), it is necessary to extract the required area and perform a plane transformation. Therefore, although the photographer can concentrate on the experience (while capturing images), the amount of work that follows becomes enormous. To address these issues, we considered a configuration like that in Example 1.

[0024] (Example 1) Figures 1A to 1D illustrate a camera system according to this embodiment, which consists of a camera body 1 including an imaging / detection unit 10 as a wearable imaging device, and a display device 800 that is configured separately from the camera body 1. In this embodiment, the camera body 1 and the display device 800 are shown as separate components, but they may be configured as an integrated unit. The user who wears the camera body 1 around their neck will be referred to as the user below.

[0025] Figure 1A is an external view of the camera body 1.

[0026] In Figure 1A, the camera body 1 comprises a shooting / detection unit 10, a battery unit 90 (power supply), a right-side connector 80R, and a left-side connector 80L. The right-side connector 80R connects the shooting / detection unit 10 and the battery unit 90 on the right side of the user's body (left side in Figure 1A). The left-side connector 80L connects the shooting / detection unit 10 and the battery unit 90 on the left side of the user's body (right side in Figure 1A).

[0027] The imaging and detection unit 10 includes a face direction detection window 13, a start switch 14, a stop switch 15, an imaging lens 16, an LED 17, and microphones 19L and 19R.

[0028] The face direction detection window 13 transmits infrared light and its reflected light emitted from an infrared LED 22 (Figure 5: infrared irradiation means) built into the shooting / detection unit 10 for detecting the position of various parts of the user's face.

[0029] The start switch 14 is a switch used to start imaging.

[0030] The stop switch 15 is a switch used to stop image acquisition.

[0031] The imaging lens 16 guides the light rays to be imaged into the imaging / detection unit 10, forming an optical image on the solid-state image sensor 42 (Figure 5).

[0032] LED17 is an LED that indicates imaging is in progress or displays a warning.

[0033] Microphones 19R and 19L are microphones that pick up ambient sounds. Microphone 19L picks up sounds from the left side of the user's surroundings (right side in Figure 1A), and microphone 19R picks up sounds from the right side of the user's surroundings (left side in Figure 1A).

[0034] Figure 1B shows the camera body 1 being worn by the user.

[0035] When the battery unit 90 is attached to the user's back and the shooting / detection unit 10 is attached to the user's front, the left and right connection parts 80L and 80R, which are connected to the left and right ends of the shooting / detection unit 10, bias and support it towards the chest. As a result, the shooting / detection unit 10 is positioned in front of the user's collarbone. At this time, the face direction detection window 13 is positioned below the user's chin. Inside the face direction detection window 13 is an infrared focusing lens 26, which will be shown later in Figure 2E. The optical axis (detection optical axis) of the infrared focusing lens 26 is directed toward the user's face and is oriented in a different direction from the optical axis (imaging optical axis) of the imaging lens 16. The face direction detection unit 20 (face direction detection means, see Figure 5), including the infrared focusing lens 26, detects the user's observation direction from the position of each part of the face. This enables imaging in that observation direction by the shooting unit 40 (imaging means), which will be described later.

[0036] Methods for adjusting the setting position due to individual differences in body shape and clothing will be discussed later.

[0037] Furthermore, by positioning the shooting / detection unit 10 on the front of the body and the battery unit 90 on the back, the weight is distributed, which reduces user fatigue and suppresses displacement caused by centrifugal force when the user moves.

[0038] In this embodiment, the image capture / detection unit 10 is shown to be mounted in a position near the user's collarbone, but this is not the only option. That is, as long as the camera body 1 can detect the user's viewing direction using the face direction detection unit 20 and capture images in that viewing direction using the image capture unit 40, the camera body 1 may be mounted anywhere on the user's body other than their head.

[0039] Figure 1C is a view of the battery unit 90 from the rear of Figure 1A.

[0040] In Figure 1C, the battery unit 90 includes a charging cable insertion port 91, adjustment buttons 92L and 92R, and a spine protection cutout 93.

[0041] The charging cable port 91 is an insertion port for a charging cable (not shown), and through this charging cable, the internal batteries 94L and 94R (see Figure 3A) are charged from an external power source, and power is supplied to the shooting / detection unit 10.

[0042] The adjustment buttons 92L and 92R are used to adjust the length of the band sections 82L and 82R of the left and right connection sections 80L and 80R. Adjustment button 92L is for adjusting the band section 82L on the left side, and adjustment button 92R is for adjusting the band section 82R on the right side. In this embodiment, the lengths of the band sections 82L and 82R are adjusted independently using adjustment buttons 92L and 92R, but it is also possible to adjust the lengths of the band sections 82L and 82R simultaneously with a single button.

[0043] The spine-relief cutout 93 is a cutout designed to avoid contact with the user's spine, preventing the battery unit 90 from touching the spine. By avoiding the protruding part of the human spine, it reduces discomfort during wear and prevents the device from moving from side to side during use.

[0044] Figure 1D is an external view of the display device 800 as a portable device according to Embodiment 1, which is configured separately from the camera body 1.

[0045] In Figure 1D, the display device 800 includes button A802, display unit 803, button B804, front camera 805, face sensor 806, angular velocity sensor 807, and acceleration sensor 808. Although not shown in Figure 1D, it also includes a wireless LAN capable of high-speed connection with the camera body 1.

[0046] Button A802 is a button that functions as the power button for the display device 800. The display device 800 accepts power ON and OFF operations by pressing and holding button A802, and accepts instructions for other processing timings by pressing button A802 briefly.

[0047] The display unit 803 allows users to view images captured by the camera body 1 and display menu screens necessary for settings. In this embodiment, a transparent touch sensor is also provided on the top surface of the display unit 803 to accept touch operations on the displayed screen (e.g., the menu screen).

[0048] Button B804 functions as a calibration button 854, which is used in the calibration process described later.

[0049] The front camera 805 is a camera capable of capturing images of a person observing the display device 800.

[0050] The face sensor 806 detects the face shape and viewing direction of a person observing the display device 800. The specific structure of the face sensor 806 is not particularly limited, but it can be implemented using various sensors such as a structural light sensor, a ToF sensor, or a millimeter-wave radar.

[0051] The angular velocity sensor 807 is located inside the display device 800 and is therefore shown with a dotted line in the perspective view. The display device 800 in this embodiment also has a calibrator function, which will be described later, and is therefore equipped with a gyro sensor in three directions: X, Y, and Z.

[0052] The accelerometer 808 detects the orientation of the display device 800.

[0053] In this embodiment, a standard smartphone is used as the display device 800, and the camera system according to the present invention can be implemented by making the firmware of the smartphone compatible with the firmware of the camera body 1. However, the camera system according to the present invention can also be implemented by making the firmware of the camera body 1 compatible with the application and OS of the smartphone used as the display device 800.

[0054] Figures 2A to 2F illustrate the imaging and detection unit 10 in detail. In subsequent figures, parts that have already been described will be numbered the same way to indicate the same function, and their explanation in this specification will be omitted.

[0055] Figure 2A is a front view of the imaging and detection unit 10.

[0056] The right-side connector 80R has a rigid angle-holding section 81R and a band section 82R that maintain the angle with the imaging / detection section 10, and the left-side connector 80L has an angle-holding section 81L and a band section 82L.

[0057] Figure 2B shows the shapes of the band sections 82L and 82R of the left and right connection sections 80L and 80R. In this figure, the angle holding sections 81L and 81R are shown transparently to illustrate the shape of the band sections 82L and 82R.

[0058] The band portion 82L includes a left-side connection surface 83L, which is positioned on the left side of the user's body (right side in Figure 2B) when the camera body 1 is attached, and an electrical cable 84. The band portion 82R includes a right-side connection surface 83R, which is positioned on the right side of the user's body (left side in Figure 2B) when the camera body 1 is attached.

[0059] The left connection surface 83L is connected to the angle holding part 81L and has a non-circular cross-sectional shape, in this case, an ellipse. The right connection surface 83R also has a similar ellipse shape. The right connection surface 83R and the left connection surface 83L are shaped like the Japanese katakana character "ハ". That is, as you move from the bottom to the top in Figure 2B, the distance between the symmetrical parts of the right connection surface 83R and the left connection surface 83L becomes smaller. As a result, when the user wears the camera body 1, the long axis directions of the left and right connection surfaces 83L and 83R are aligned with the user's body, resulting in a comfortable fit when the band parts 82L and 82R touch the user's body, and preventing the shooting / detection unit 10 from moving in the left, right, front, or back directions.

[0060] The electrical cable 84 (power supply means) is wired inside the band section 82L and electrically connects the battery section 90 and the imaging / detection section 10. The electrical cable 84 connects the power supply of the battery section 90 to the imaging / detection section 10 and also transmits and receives electrical signals to and from the outside.

[0061] Figure 2C shows the imaging / detection unit 10 viewed from the back. Since Figure 2C is a view from the side that contacts the user's body, i.e., the opposite side of Figure 2A, the positional relationship between the right connection part 80R and the left connection part 80L is reversed compared to Figure 2A.

[0062] The imaging and detection unit 10 is equipped with a power switch 11, an imaging mode switch 12, and chest connection pads 18a and 18b on its back side.

[0063] The power switch 11 is a power switch that switches the power of the camera body 1 ON / OFF. In this embodiment, the power switch 11 is a slide lever type switch, but is not limited to this. For example, the power switch 11 may be a push type switch, or it may be a switch integrated with a slide cover (not shown) of the imaging lens 16.

[0064] The imaging mode switch 12 (changing means) is a switch that changes the imaging mode and can change the mode related to imaging. In this embodiment, the imaging mode switch 12 can switch to still image mode, video mode, and a pre-setting mode set using the display device 800, which will be described later. In this embodiment, the imaging mode switch 12 is a switch in the form of a slide lever that allows selection of one of "Photo," "Normal," or "Pre" as shown in Figure 2C by sliding the lever. The imaging mode changes to still image mode by sliding to "Photo," to video mode by sliding to "Normal," and to pre-setting mode by sliding to "Pre." Note that the imaging mode switch 12 is not limited to the form of this embodiment as long as it is a switch that can change the imaging mode. For example, the imaging mode switch 12 may be composed of three buttons: "Photo," "Normal," and "Pre."

[0065] The chest connection pads 18a and 18b (fixing means) are the parts that come into contact with the user's body when the imaging / detection unit 10 is biased against the user's body. As shown in Figure 2A, the imaging / detection unit 10 is shaped so that its horizontal (left-right) length is longer than its vertical (up-down) length when attached, and the chest connection pads 18a and 18b are positioned near the left and right ends of the imaging / detection unit 10. This arrangement makes it possible to suppress lateral rotational blur during imaging with the camera body 1. In addition, the presence of the chest connection pads 18a and 18b prevents the power switch 11 and the imaging mode switch 12 from coming into contact with the body. Furthermore, the chest connection pads 18a and 18b also serve to prevent heat from being transferred to the user's body even if the temperature of the imaging / detection unit 10 rises during long-term imaging, and also play a role in adjusting the angle of the imaging / detection unit 10.

[0066] Figure 2D is a top view of the imaging and detection unit 10.

[0067] As shown in Figure 2D, a face direction detection window 13 is provided in the center of the upper surface of the imaging / detection unit 10, and the chest connection pads 18a and 18b protrude from the imaging / detection unit 10.

[0068] Figure 2E shows the configuration of the face direction detection unit 20, which is located inside the imaging and detection unit 10 and positioned below the face direction detection window 13.

[0069] The face direction detection unit 20 includes an infrared LED 22 and an infrared focusing lens 26. The face direction detection unit 20 further includes an infrared LED lighting circuit 21 and an infrared detection processing device 27, which will be described later in Figure 5.

[0070] The infrared LED 22 emits infrared light 23 (Figure 5) towards the user.

[0071] The infrared focusing lens 26 is a lens that focuses the reflected light rays 25 (Figure 5) reflected from the user when infrared rays 23 are emitted from the infrared LED 22 onto a sensor (not shown) of the infrared detection processing device 27.

[0072] Figure 2F shows the camera body 1 as seen from the left side of the user's body while the user is wearing it.

[0073] The angle adjustment button 85L is located on the angle holding unit 81L and is used to adjust the angle of the shooting / detection unit 10. Although not shown in this figure, an angle adjustment button is also located inside the angle holding unit 81R on the opposite side, in a position symmetrical to the angle adjustment button 85L.

[0074] The angle adjustment button is visible in Figures 2A, 2C, and 2D, but it has been omitted for the sake of simplicity in this explanation.

[0075] The user can change the angle between the imaging / detection unit 10 and the angle holding unit 81L by pressing the angle adjustment button 85L and moving the angle holding unit 81L up and down in the direction of Figure 2F. The same applies to the right side. In addition, the chest connection pads 18a and 18b can have their protrusion angles changed. Through the action of these two types of angle-changing members (angle adjustment button and chest connection pads), the imaging / detection unit 10 can adjust the optical axis of the imaging lens 16 horizontally regardless of individual differences in the shape of the user's chest position.

[0076] Figure 3 is a diagram illustrating the details of the battery unit 90.

[0077] Figure 3(a) is a view of the battery unit 90 from the rear, with a partial perspective.

[0078] As shown in Figure 3(a), the battery unit 90 has two batteries, a left battery 94L and a right battery 94R, symmetrically mounted inside to balance its weight. By arranging the left and right batteries 94L and 94R symmetrically with respect to the center of the battery unit 90 in this way, the weight balance on the left and right sides is adjusted, preventing the camera body 1 from shifting position. The battery unit 90 may also be configured to house only one battery.

[0079] Figure 3(b) is a view of the battery section 90 from above. In this figure as well, batteries 94L and 94R are shown in perspective.

[0080] As shown in Figure 3(b), by symmetrically arranging the batteries 94L and 94R on both sides of the spine-guard cutout 93, it is possible to attach the relatively heavy battery unit 90 to the user without causing any burden.

[0081] Figure 3(c) is a view of the battery unit 90 from the back. Figure 3(c) is a view from the side that comes into contact with the user's body, that is, the opposite side from Figure 3(a).

[0082] As shown in Figure 3(c), the spine-guard cutout 93 is positioned in the center to follow the user's spine.

[0083] Figure 4 is a functional block diagram of the camera body 1. Details will be explained later, but here we will use Figure 4 to describe the general flow of processing performed by the camera body 1.

[0084] In Figure 4, the camera body 1 comprises a face direction detection unit 20, a recording direction / angle determination unit 30, a shooting unit 40, an image cropping / development processing unit 50, a primary recording unit 60, a transmission unit 70, and other control units 111. These functional blocks are executed under the control of the overall control CPU 101 (Figure 5), which controls the entire camera body 1.

[0085] The face direction detection unit 20 (observation direction detection means) is a functional block executed by the infrared LED 22 and infrared detection processing device 27 mentioned above, which detects the face direction, infers the observation direction, and passes this to the recording direction / angle determination unit 30.

[0086] The recording direction / angle determination unit 30 (recording direction determination means) performs various calculations based on the observation direction inferred by the face direction detection unit 20 to determine the position and range information for extracting the image from the shooting unit 40, and passes this information to the image extraction / development processing unit 50.

[0087] The shooting unit 40 converts the light rays from the subject into a wide-angle image and passes that image to the image extraction and development processing unit 50.

[0088] The image extraction and development processing unit 50 (development means) uses information from the recording direction and field of view determination unit 30 to extract and develop only the image in the direction the user is looking from the image from the shooting unit 40, and then passes it to the primary recording unit 60.

[0089] The primary recording unit 60 is a functional block consisting of a primary memory 103 (Figure 5) and the like, which records video information and transmits it to the transmission unit 70 at the necessary timing.

[0090] The transmitting unit 70 (video output means) wirelessly connects to predetermined communication partners, namely the display device 800 (Figure 1D), the calibrator 850, and the simple display device 900, and communicates with them.

[0091] The display device 800 can connect to the transmission unit 70 via a high-speed wireless LAN (hereinafter referred to as "high-speed wireless"). In this embodiment, wireless communication corresponding to the IEEE 802.11ax (WiFi 6) standard is used for the high-speed wireless, but other standards, such as WiFi 4 or WiFi 5, may also be used. Furthermore, the display device 800 may be a device developed specifically for the camera body 1, or it may be a general-purpose smartphone or tablet device.

[0092] Furthermore, the connection between the transmitter 70 and the display device 800 may use low-power wireless communication, or it may be connected using both high-speed and low-power wireless communication, or switched between the two. In this embodiment, data with a large amount of data, such as video files of video footage described later, is transmitted using high-speed wireless communication, while lightweight data or data that can take longer to transmit is transmitted using low-power wireless communication. In this embodiment, Bluetooth is used for low-power wireless communication, but other short-range wireless communication such as NFC (Near Field Communication) may also be used.

[0093] The calibrator 850 is a device used for initial setup of the camera body 1 and for individual settings, and like the display device 800, it can connect to the transmitter 70 via high-speed wireless. Further details about the calibrator 850 will be described later. The display device 800 may also incorporate the functions of the calibrator 850.

[0094] The simplified display device 900 is a display device that can only be connected to the transmitter 70 via, for example, low-power wireless communication.

[0095] The simplified display device 900 cannot transmit video footage to the transmission unit 70 due to time constraints, but it can transmit the timing of the start and stop of image capture, and perform image confirmation to the extent of composition. Furthermore, the simplified display device 900, like the display device 800, may be a device developed specifically for the camera body 1, or it may be a smartwatch or similar device.

[0096] Figure 5 is a block diagram showing the hardware configuration of camera body 1. Furthermore, the same numbers are used for the configurations and functions described using Figures 1A to 1C, etc., and detailed explanations are omitted.

[0097] In Figure 5, the camera body 1 includes an overall control CPU 101, a power switch 11, an imaging mode switch 12, a face direction detection window 13, a start switch 14, a stop switch 15, an imaging lens 16, and an LED 17.

[0098] The camera body 1 also includes an infrared LED lighting circuit 21, an infrared LED 22, an infrared focusing lens 26, and an infrared detection processing device 27, which constitute a face direction detection unit 20 (Figure 4).

[0099] Furthermore, the camera body 1 includes an imaging unit 40 (Figure 4) consisting of an imaging driver 41, a solid-state image sensor 42, and an imaging signal processing circuit 43, and a transmitting unit 70 (Figure 4) consisting of a low-power wireless unit 71 and a high-speed wireless unit 72.

[0100] In this embodiment, the camera body 1 is provided with only one shooting unit 40, but two or more shooting units 40 may be provided to capture 3D images, capture images with a wider angle of view than that obtainable with one shooting unit 40, or capture images in multiple directions.

[0101] The camera body 1 also includes various types of memory, such as a large-capacity non-volatile memory 51, a built-in non-volatile memory 102, and a primary memory 103.

[0102] Furthermore, the camera body 1 includes an audio processing unit 104, a speaker 105, a vibrator 106, an angular velocity sensor 107, an acceleration sensor 108, and various switches 110.

[0103] The overall control CPU 101, to which the aforementioned power switch 11 and other components are connected as shown in Figure 2C, controls the camera body 1. The recording direction / angle determination unit 30, image cropping / development processing unit 50, and other control units 111 shown in Figure 4 are implemented by the overall control CPU 101 itself.

[0104] The infrared LED lighting circuit 21 controls the on / off state of the infrared LED 22 as described above using Figure 2E, and controls the emission of infrared light 23 from the infrared LED 22 toward the user.

[0105] The face direction detection window 13 is composed of a visible light cut filter, which blocks almost all visible light but allows sufficient transmission of infrared light 23 and its reflected light 25, which are in the infrared region.

[0106] The infrared focusing lens 26 is a lens that focuses reflected light rays 25.

[0107] The infrared detection processing unit 27 (infrared detection means) has a sensor that detects reflected light rays 25 focused by an infrared focusing lens 26. This sensor converts the image formed by the focused reflected light rays 25 into sensor data and passes it to the overall control CPU 101.

[0108] As shown in Figure 1B, when the user is wearing the camera body 1, the face direction detection window 13 is located below the user's chin. Therefore, as shown in Figure 5, the infrared light 23 emitted from the infrared LED 22 passes through the face direction detection window 13 and is irradiated onto the infrared irradiation surface 24, which is near the user's chin. The reflected light 25 reflected from the infrared irradiation surface 24 also passes through the face direction detection window 13 and is focused by the infrared condensing lens 26 onto the sensor in the infrared detection processing device 27.

[0109] The various switches 110 are not shown in Figures 1A to 1C, etc., and although details are omitted, they are switches that perform functions unrelated to this embodiment.

[0110] The imaging driver 41 includes a timing generator and other components, and generates and outputs various timing signals to each part involved in imaging, thereby driving the solid-state image sensor 42.

[0111] The solid-state image sensor 42 outputs a signal obtained by photoelectric conversion of the subject image projected from the imaging lens 16, as explained using Figure 1A, to the imaging signal processing circuit 43.

[0112] The imaging signal processing circuit 43 performs processing such as clamping and A / D conversion on the signal from the solid-state image sensor 42 and outputs the generated imaging data to the overall control CPU 101.

[0113] The built-in non-volatile memory 102 uses flash memory or the like and stores the startup program for the overall control CPU 101 and the settings for various program modes. In this embodiment, the observation field of view (angle of view) and the effectiveness level of vibration control can be set, so these settings are also recorded.

[0114] The primary memory 103 consists of RAM and other components, and temporarily stores video data being processed, as well as the calculation results of the overall control CPU 101.

[0115] The large-capacity non-volatile memory 51 stores image data. In this embodiment, the large-capacity non-volatile memory 51 is a non-removable semiconductor memory. However, the large-capacity non-volatile memory 51 may be configured as a removable recording medium such as an SD card, or it may be used in combination with the built-in non-volatile memory 102.

[0116] The low-power wireless unit 71 exchanges data with the display device 800, calibrator 850, and simple display device 900 using low-power wireless communication.

[0117] The high-speed wireless unit 72 exchanges data with the display device 800 and the calibrator 850 using high-speed wireless communication.

[0118] The audio processing unit 104 processes the external sound (analog signal) picked up by microphones 19L and 19R and generates an audio signal.

[0119] The LED 17, speaker 105, and vibrator 106 communicate or warn the user about the status of the camera body 1 by emitting light, sound, or vibrating.

[0120] The angular velocity sensor 107 is a sensor that uses a gyroscope or the like, and detects the movement of the camera body 1 itself as gyro data.

[0121] The acceleration sensor 108 detects the orientation of the shooting / detection unit 10.

[0122] Figure 6 is a block diagram showing the hardware configuration of the display device 800. For the sake of simplicity, the same reference numerals are used for parts that were explained using Figure 1D, and their explanations are omitted.

[0123] In Figure 6, the display device 800 includes a display device control unit 801, button A 802, display unit 803, button B 804, face sensor 806, angular velocity sensor 807, acceleration sensor 808, image capture signal processing circuit 809, and various switches 811.

[0124] The display device 800 also includes a built-in non-volatile memory 812, a primary memory 813, a large-capacity non-volatile memory 814, a speaker 815, a vibrator 816, an LED 817, an audio processing unit 820, a low-power wireless unit 871, and a high-speed wireless unit 872. Each of these elements is connected to the display device control unit 801.

[0125] The display device control unit 801 is configured with a CPU and controls the display device 800.

[0126] The imaging signal processing circuit 809 performs the same functions as the imaging driver 41, solid-state image sensor 42, and imaging signal processing circuit 43 inside the camera body 1, and together with the in-camera lens 805a, it constitutes the in-camera 805 shown in Figure 1D. The data output by the imaging signal processing circuit 809 is processed in the display device control unit 801. The details of this data processing will be described later.

[0127] The various switches 811 are switches that perform functions unrelated to this embodiment.

[0128] The angular velocity sensor 807 is a sensor that uses a gyroscope or the like, and detects the movement of the display device 800 itself.

[0129] The accelerometer 808 detects the orientation of the display device 800 itself.

[0130] The built-in non-volatile memory 812 uses flash memory or similar technology and stores the startup program for the display device control unit 801 and the settings for various program modes.

[0131] The primary memory 813 is composed of RAM or the like and temporarily stores video data being processed and the calculation results of the imaging signal processing circuit 809. In this embodiment, during video recording, gyro data detected by the angular velocity sensor 107 at the imaging time of each frame is associated with each frame and stored in the primary memory 813.

[0132] The high-capacity non-volatile memory 814 stores image data from the display device 800. In this embodiment, the high-capacity non-volatile memory 814 is configured as a removable memory, like an SD card. Alternatively, it may be configured as a non-removable memory, like the high-capacity non-volatile memory 51 located in the camera body 1.

[0133] The speaker 815, vibrator 816, and LED 817 communicate the status of the display device 800 to the user or provide warnings by emitting sound, vibrating, or emitting light.

[0134] The audio processing unit 820 processes external sounds (analog signals) picked up by the left microphone 819L and the right microphone 819R, and generates an audio signal.

[0135] The low-power wireless unit 871 exchanges data with the camera body 1 using low-power wireless communication.

[0136] The high-speed wireless unit 872 exchanges data with the camera body 1 using high-speed wireless communication.

[0137] The face sensor 806 (face detection means) includes an infrared LED lighting circuit 821, an infrared LED 822, an infrared focusing lens 826, and an infrared detection processing device 827.

[0138] The infrared LED lighting circuit 821 is a circuit that has the same function as the infrared LED lighting circuit 21 in Figure 5, and controls the lighting and extinguishing of the infrared LED 822 and controls the emission of infrared light 823 from the infrared LED 822 toward the user.

[0139] The infrared focusing lens 826 is a lens that focuses the reflected light rays 825 of the infrared rays 823.

[0140] The infrared detection and processing unit 827 has a sensor that detects reflected light rays focused by the infrared focusing lens 826. This sensor converts the focused reflected light rays 825 into sensor data and passes it to the display device control unit 801.

[0141] When the face sensor 806 shown in Figure 1D is pointed at the user, as shown in Figure 6, infrared light 823 emitted from the infrared LED 822 is irradiated onto the infrared irradiation surface 824, which is the entire face of the user. The reflected light rays 825 reflected from the infrared irradiation surface 824 are then focused by the infrared focusing lens 826 onto the sensor in the infrared detection processing device 827.

[0142] The other function unit 830 performs smartphone functions such as telephone functions that are not related to this embodiment.

[0143] The following explains how to use the camera body 1 and the display device 800.

[0144] Figure 7A is a flowchart showing an overview of the image recording process according to this embodiment, which is performed in the camera body 1 and the display device 800.

[0145] For further explanation, Figure 7A indicates on the right side of each step which device shown in Figure 4 is performing that step. Specifically, steps S100 to S700 in Figure 7A are performed by the camera body 1, and steps S800 to S1000 in Figure 7A are performed by the display device 800.

[0146] When the power switch 11 is turned ON and power is supplied to the camera body 1, the overall control CPU 101 starts up and reads the startup program from the built-in non-volatile memory 102. Then, in step S100, the overall control CPU 101 performs preparatory operations to configure the camera body 1 before image capture. Details of the preparatory operations will be described later using Figure 7B.

[0147] In step S200, the face direction detection unit 20 detects the face direction and performs a face direction detection process to infer the observation direction. Details of the face direction detection process will be described later using Figure 7C. This process is executed at a predetermined frame rate.

[0148] In step S300, the recording direction / angle determination unit 30 performs the recording direction / range determination process. Details of the recording direction / range determination process will be described later using Figure 7D.

[0149] In step S400, the imaging unit 40 performs imaging and generates imaging data.

[0150] In step S500, the image extraction and development processing unit 50 uses the recording direction and field of view information determined in step S300 to extract the image from the imaging data generated in step S400 and performs recording range development processing on that area. Details of the recording range development processing will be described later with reference to Figure 7E.

[0151] In step S600, the primary recording unit 60 (video recording means) performs a primary recording process in which the video developed in step S500 is saved as video data in the primary memory 103. Details of the primary recording process will be described later with reference to Figure 14.

[0152] In step S700, the transmission unit 70 performs a transfer process to the display device 800, in which it wirelessly transmits the video recorded in step S600 to the display device 800 at a specified timing. Details of the transfer process to the display device 800 will be described later with reference to Figure 16.

[0153] Steps from step S800 onward are executed on the display device 800.

[0154] In step S800, the display device control unit 801 performs optical correction processing to correct the optical aberrations of the video transferred from the camera body 1 in step S700. Details of the optical correction processing will be described later with reference to Figure 17.

[0155] In step S900, the display device control unit 801 performs vibration damping on the image that was optically corrected in step S800. Details of the vibration damping process will be described later with reference to Figure 19.

[0156] Furthermore, the order of steps S800 and S900 can be reversed. In other words, you can perform image stabilization first and then optical correction afterward.

[0157] In step S1000, the display device control unit 801 (video recording means) performs secondary recording, recording the video, which has undergone optical correction processing and vibration damping processing in steps S800 and S900, into the large-capacity non-volatile memory 814, and then terminates this process.

[0158] Next, using Figures 7B to 7F, we will explain in detail the subroutines for each step described in Figure 7A, along with the order of processing, using other diagrams as well.

[0159] Figure 7B is a flowchart of the subroutine for the preparation operation process in step S100 of Figure 7A. This process will be explained below using the parts illustrated in Figures 2 and 5.

[0160] In step S101, it is determined whether the power switch 11 is ON or OFF. If the power remains OFF, the system waits; if it turns ON, the system proceeds to step S102.

[0161] In step S102, the mode selected by the imaging mode switch 12 is determined. As a result of the determination, if the mode selected by the imaging mode switch 12 is the video mode, the process proceeds to step S103.

[0162] In step S103, various settings for the video mode are read from the built-in non-volatile memory 102 and stored in the primary memory 103, and then the process proceeds to step S104. Here, the various settings for the video mode include the angle-of-view setting value V (predetermined to be 90° in this embodiment) and the anti-shake level specified by "strong", "medium", "off", etc.

[0163] In step S104, after starting the operation of the imaging driver 41 for the video mode, this subroutine is exited.

[0164] As a result of the determination in step S102, if the mode selected by the imaging mode switch 12 is the still image mode, the process proceeds to step S106.

[0165] In step S106, various settings for the still image mode are read from the built-in non-volatile memory 102 and stored in the primary memory 103, and then the process proceeds to step S107. Here, the various settings for the still image mode include the angle-of-view setting value V (predetermined to be 45° in this embodiment) and the anti-shake level specified by "strong", "medium", "off", etc.

[0166] In step S107, after starting the operation of the imaging driver 41 for the still image mode, this subroutine is exited.

[0167] If, as a result of the determination in step S102, the mode selected by the imaging mode switch 12 is the preset mode, the process proceeds to step S108. Here, the preset mode is a mode in which the imaging mode is set for the camera body 1 from an external device such as the display device 800, and is one of the three imaging modes that can be switched by the imaging mode switch 12. The preset mode is, in other words, a mode for custom shooting. Here, since the camera body 1 is a small wearable device, an operation switch or a setting screen for changing its detailed settings is not provided on the camera body 1, and the detailed settings of the camera body 1 are changed by an external device such as the display device 800.

[0168] For example, even in the same video imaging, consider the case where you want to continuously image with a 90° angle of view and a 110° angle of view. Since a 90° angle of view is set in the normal video mode, when performing such imaging, first, after imaging in the normal video mode, the video imaging is terminated once, the setting screen of the camera body 1 is displayed on the display device 800, and an operation to switch the angle of view to 110° is required. However, if it is during some event, such an operation on the display device 800 is troublesome.

[0169] On the other hand, if the preset mode is pre-set to a mode for video imaging with an angle of view of 110°, after the video imaging with an angle of view of 90° is completed, simply sliding the imaging mode switch 12 to "Pre" allows for an immediate change to video imaging with an angle of view of 110°. That is, the user does not need to interrupt the current action and perform the above-mentioned troublesome operation.

[0170] In addition, the content set in the preset mode may include not only the angle of view but also the anti-shake level specified by "strong", "medium", "off", etc., and settings for voice recognition not described in this embodiment.

[0171] In step S108, the various settings for the pre-setting mode are read from the built-in non-volatile memory 102 and saved in the primary memory 103, after which the process proceeds to step S109. Here, the various settings for the pre-setting mode include the field of view setting value V and the image stabilization level specified as "strong," "medium," or "off."

[0172] In step S109, the operation of the imaging driver 41 for the pre-setting mode is started, and then the subroutine is exited.

[0173] Here, we will explain the various video mode settings read in step S103 using Figure 13.

[0174] Figure 13 shows the menu screen for various video mode settings displayed on the display unit 803 of the display device 800 before image capture by the camera body 1. Note that the same reference numerals are used for parts identical to those in Figure 1D, and their explanation is omitted. The display unit 803 has a touch panel function, and the following explanation assumes that it functions via touch operations, including swiping.

[0175] In Figure 13, the menu screen includes a preview screen 831, a zoom lever 832, a recording start / stop button 833, a switch 834, a battery level indicator 835, a button 836, a lever 837, and an icon display unit 838.

[0176] The preview screen 831 allows you to check the image captured by the camera body 1, and to check the zoom level and field of view.

[0177] The zoom lever 832 is an operating unit that allows zoom settings to be adjusted by shifting it left or right. In this embodiment, we will describe a case where four values, 45°, 90°, 110°, and 130°, can be set as the angle of view setting value V, but the zoom lever 832 may also be used to set values ​​other than these as the angle of view setting value V.

[0178] The recording start / stop button 833 is a toggle switch that combines the functions of both the start switch 14 and the stop switch 15.

[0179] Switch 834 is a switch that toggles the vibration damping on and off.

[0180] Battery level indicator 835 displays the remaining battery level of the camera body 1.

[0181] Button 836 is used to change the mode.

[0182] Lever 837 is a lever for setting the vibration isolation level. In this embodiment, only "strong" and "medium" vibration isolation levels can be set, but other vibration isolation levels, such as "weak," may also be set. Alternatively, the vibration isolation level may be set steplessly.

[0183] The icon display unit 838 displays multiple thumbnail icons for previewing.

[0184] Figure 7C is a flowchart of the subroutine for the face direction detection process in step S200 of Figure 7A. Before explaining the details of this process, we will explain the method of detecting face direction using infrared projection using Figures 8A to 8K.

[0185] Figure 8A shows a visible light image of the user's face as seen from the position of the face direction detection window 13.

[0186] The image in Figure 8A is identical to the image captured by a visible light image sensor when the face direction detection window 13 does not have a visible light cut filter component, transmits sufficient visible light, and the infrared detection processing device 27 is a visible light image sensor.

[0187] The video in Figure 8A shows the user's face, including the front of the neck above the collarbone 201, the base of the jaw 202, the tip of the chin 203, and the nose 204.

[0188] Figure 8B shows the case where the fluorescent light 205 in the room is reflected as a background in the visible light image of the user shown in Figure 8A.

[0189] In the visible light image of FIG. 8B, a plurality of fluorescent lamps 205 around the user are shown. Since various backgrounds and the like are reflected in the user's image depending on the usage conditions, it becomes difficult for the face direction detection unit 20 or the overall control CPU 101 to separate the image of the face from the visible light image. On the other hand, although there is a technology for separating such images by using AI or the like, a high level of ability is required for the overall control CPU 101, which is not suitable for the camera body 1 which is a portable device.

[0190] Therefore, the camera 1 of the first embodiment detects the user's face using an infrared light image. Since the face direction detection window 13 is composed of a visible light cut filter and thus visible light hardly passes through, the image of the infrared detection processing device 27 is not like the images in FIGS. 8A and 8B.

[0191] FIG. 8C is a diagram showing an infrared light image when the user shown in FIG. 8B and the fluorescent lamp as the background thereof are imaged by the sensor of the infrared detection processing device 27 through the face direction detection window 13 with the infrared LED 22 turned off.

[0192] In the infrared light image of FIG. 8C, the user's neck and jaw are dark. On the other hand, since the fluorescent lamp 205 has not only visible light components but also infrared components, it appears slightly brighter.

[0193] FIG. 8D is a diagram showing an image when the user shown in FIG. 8B and the fluorescent lamp as the background thereof are imaged by the sensor of the infrared detection processing device 27 through the face direction detection window 13 with the infrared LED 22 turned on.

[0194] In the image of FIG. 8D, the user's neck and jaw are bright. On the other hand, unlike FIG. 8C, the brightness around the fluorescent lamp 205 has not changed.

[0195] FIG. 8E is a diagram showing a difference image calculated by subtracting the image of FIG. 8C from the image of FIG. 8D. It can be seen that the user's face stands out.

[0196] In this way, the overall control CPU 101 (image acquisition means) calculates the difference between the images formed by the infrared detection processing device 27's sensor when the infrared LED 22 is lit and when it is not, thereby obtaining a difference image (hereinafter also referred to as a face image) in which the user's face is extracted.

[0197] In this embodiment, the face direction detection unit 20 employs a method of acquiring face images by extracting infrared reflection intensity as a two-dimensional image using the infrared detection processing unit 27. The sensor of the infrared detection processing unit 27 employs a structure similar to that of a general image sensor and acquires face images one frame at a time. The vertical synchronization signal (hereinafter referred to as the V signal) for frame synchronization is generated by the infrared detection processing unit 27 and output to the overall control CPU 101.

[0198] Figure 9 is a timing chart showing the timing of the infrared LED 22 turning on and off and the associated signals.

[0199] Figure 9(a) shows the timing at which the V signal is generated by the infrared detection processing unit 27. When the V signal becomes Hi, the timing of frame synchronization and the on / off of the infrared LED 22 is determined.

[0200] In Figure 9(a), t1 represents the period for the first facial image acquisition, and t2 represents the period for the second facial image acquisition. Figures 9(a), (b), (c), and (d) are presented so that their horizontal time axes are identical.

[0201] Figure 9(b) shows the H position of the image signal output from the sensor of the infrared detection processing unit 27 on the vertical axis. The infrared detection processing unit 27 controls the movement of its sensor so that the H position of the image signal is synchronized with the V signal, as shown in Figure 9(b). As mentioned above, the sensor of the infrared detection processing unit 27 employs a structure similar to that of a general image sensor, and its movement is well known, so the detailed control is omitted.

[0202] Figure 9(c) shows the switching timing between Hi and Low of the IR-ON signal output from the overall control CPU 101 to the infrared LED lighting circuit 21. The switching between Hi and Low of the IR-ON signal is controlled by the overall control CPU 101 in synchronization with the V signal, as shown in Figure 9(c). Specifically, the overall control CPU 101 outputs a Low IR-ON signal to the infrared LED lighting circuit 21 during period t1, and outputs a Hi IR-ON signal to the infrared LED lighting circuit 21 during period t2.

[0203] Here, while the IR-ON signal is Hi, the infrared LED lighting circuit 21 lights up the infrared LED 22, and infrared light 23 is projected onto the user. On the other hand, while the IR-ON signal is Low, the infrared LED lighting circuit 21 turns off the infrared LED 22.

[0204] Figure 9(d) shows the imaging data output from the infrared detection and processing unit 27 sensor to the overall control CPU 101. The vertical axis represents the signal intensity and indicates the amount of reflected light 25 received. In other words, during period t1, the infrared LED 22 is off, so there is no reflected light 25 from the user's face, and imaging data like that shown in Figure 8C is obtained. On the other hand, during period t2, the infrared LED 22 is on, so there is reflected light 25 from the user's face, and imaging data like that shown in Figure 8D is obtained. Therefore, as shown in Figure 9(d), the signal intensity during period t2 is higher than the signal intensity during period t1 by the amount of reflected light 25 from the user's face.

[0205] Figure 9(e) shows the result of subtracting the imaging data from period t1 from the imaging data from period t2 in Figure 9(d), resulting in facial image data in which only the component of reflected light 25 from the user's face is extracted, as shown in Figure 8E.

[0206] Figure 7C shows the face direction detection process in step S200, including the operations described using Figures 8C to 8E and Figure 9 above.

[0207] First, in step S201, when the V signal output from the infrared detection processing device 27 becomes Hi, the timing V1 of the start of period t1 is acquired. Once timing V1 is acquired, the process proceeds to step S202.

[0208] Next, in step S202, the IR-ON signal is set to Low and output to the infrared LED lighting circuit 21. As a result, the infrared LED 22 is turned off.

[0209] In step S203, the imaging data for one frame output from the infrared detection processing device 27 during the period t1 is read out and temporarily stored as Frame1 in the primary memory 103.

[0210] In step S204, when the V signal output from the infrared detection processing unit 27 reaches timing V2, which marks the start of period t2, the process proceeds to step S205.

[0211] In step S205, the IR-ON signal is set to Hi and output to the infrared LED lighting circuit 21. As a result, the infrared LED 22 lights up.

[0212] In step S206, the imaging data for one frame output from the infrared detection processing unit 27 during the period t2 is read out and temporarily stored as Frame2 in the primary memory 103.

[0213] In step S207, the IR-ON signal is set to Low and output to the infrared LED lighting circuit 21. This turns off the infrared LED 22.

[0214] In step S208, Frame1 and Frame2 are read from the primary memory 103, and the difference obtained by subtracting Frame1 from Frame2 is used to calculate the light intensity Fn of the 25 components of the user's reflected light that corresponds to the face image in Figure 9(e) (this is generally known as the blackout process).

[0215] In step S209, the neck position (center of neck rotation) is extracted from the light intensity Fn.

[0216] First, the overall control CPU 101 (division means) divides the face image into multiple distance areas, which will be explained using Figure 8F, based on the light intensity Fn.

[0217] Figure 8F shows the case where the grayscale of the difference image in Figure 8E is adjusted to match the light intensity of the reflected light rays 25 of the infrared 23 projected onto the user's face and neck, in order to see the distribution of light intensity for each part of the user's face and neck.

[0218] Figure 8F(a) shows the distribution of light intensity of reflected light rays 25 in the facial image of Figure 8E, divided into regions and indicated in gray. The Xf axis is taken in the direction from the center of the user's neck to the chin for explanatory purposes.

[0219] Figure 8F(i) shows the light intensity on the Xf axis of Figure 8F(a) on the horizontal axis, and the Xf axis on the vertical axis. The light intensity increases as you move to the right on the horizontal axis.

[0220] In Figure 8F(a), the facial image is divided into six regions (distance areas) 211-216 according to light intensity.

[0221] Region 211 is the region with the strongest light intensity and is shown in white as a gray area.

[0222] Region 212 is a region where the light intensity is slightly lower than that of region 211, and is shown as a gray area, specifically in a fairly light gray color.

[0223] Region 213 is a region where the light intensity is even lower than that of region 212, and is shown as a gray area, represented by a light gray color.

[0224] Region 214 is a region where the light intensity is even lower than that of region 213, and is shown as an intermediate gray color, representing a gray range.

[0225] Region 215 is a region where the light intensity is even lower than that of region 214, and is shown as a gray area, represented by a slightly darker gray color.

[0226] Region 216 is the region with the weakest light intensity and is the darkest shade of gray. Above region 216, there is no light intensity and it is black.

[0227] This light intensity will be explained in detail below using Figure 10.

[0228] Figure 10 illustrates the vertical movement of the user's face, showing the user's position as observed from the left side.

[0229] Figure 10(a) shows the user facing forward. The imaging and detection unit 10 is located in front of the user's collarbone. In addition, infrared light 23 from the infrared LED 22 is irradiated onto the lower part of the user's head from the face direction detection window 13 located above the imaging and detection unit 10. If we let Dn be the distance from the face direction detection window 13 to the base of the neck 200 above the user's collarbone, Db be the distance from the face direction detection window 13 to the base of the chin 202, and Dc be the distance from the face direction detection window 13 to the tip of the chin 203, then it can be seen that the distances increase in the order of Dn, Db, and Dc. Since light intensity is inversely proportional to the square of the distance, the light intensity when the reflected light 25 from the infrared irradiation surface 24 is imaged by the sensor of the infrared detection processing device 27 decreases in the order of the neck 200, the base of the chin 202, and the tip of the chin 203. Furthermore, it can be seen that the light intensity of the face 204, including the nose, which is located at a distance greater than Dc from the face direction detection window 13, becomes even dimmer. In other words, in a case like Figure 10(a), it can be seen that an image with the light intensity distribution shown in Figure 8F is acquired.

[0230] Furthermore, the configuration of the face direction detection unit 20 is not limited to the configuration shown in this embodiment, as long as the direction of the user's face can be detected. For example, an infrared pattern may be irradiated from the infrared LED 22 (infrared pattern irradiation means), and the infrared pattern reflected from the irradiated object may be detected by the sensor (infrared pattern detection means) of the infrared detection processing device 27. In this case, the sensor of the infrared detection processing device 27 is preferably a structural light sensor. Alternatively, the sensor of the infrared detection processing device 27 may be a sensor that performs phase comparison between infrared rays 23 and reflected light rays 25 (infrared phase comparison means), for example, a ToF sensor.

[0231] Next, using Figure 8G, we will explain the extraction of the neck position in step S209 of Figure 7C.

[0232] Figure 8G(a) is a diagram in which the symbols indicating the various parts of the user's body in Figure 10(a), as well as the symbols for the double circle indicating the neck position and the black circle indicating the chin position, are superimposed on Figure 8F.

[0233] The white area 211 corresponds to the neck 200 (Figure 10(a)), the fairly light gray area 212 corresponds to the front of the neck 201 (Figure 10(a)), and the light gray area 213 corresponds to the base of the chin 202 (Figure 10(a)). The medium gray area 214 corresponds to the tip of the chin 203 (Figure 10(a)), and the slightly darker gray area 215 corresponds to the lower part of the face 204 (Figure 10(a)), specifically the lips and the surrounding lower face. Furthermore, the darker gray area 216 corresponds to the upper part of the face 204 (Figure 10(a)), specifically the nose and the surrounding upper face.

[0234] Furthermore, as shown in Figure 10(a), the distance between Db and Dc is small compared to the distance from the face direction detection window 13 to other parts of the user, so the difference in reflected light intensity between the light gray region 213 and the medium gray region 214 is also small.

[0235] On the other hand, as shown in Figure 10(a), the distance Dn is the shortest distance among the distances from the face direction detection window 13 to each part of the user, so the white area 211 corresponding to the neck 200 is the area with the strongest reflectivity.

[0236] Therefore, the overall control CPU 101 (setting means) sets the position 206, indicated by the double circle in Figure 8G(a), which is the center of the left and right sides of region 211 and closest to the imaging / detection unit 10, as the neck rotation center position (hereinafter referred to as neck position 206). This process is what is done in step S209 of Figure 7C.

[0237] Next, using Figure 8G, we will explain the extraction of the chin position in step S210 of Figure 7C.

[0238] As shown in Figure 8G(a), the intermediate gray region 214, which is brighter than the region 215 corresponding to the lower part of the face including the lips within the face 204, is the region including the chin. As can be seen in Figure 8G(b), the rate of change in distance from the face direction detection window 13 is large, so the light intensity drops sharply in region 215 adjacent to region 214. The overall control CPU 101 determines that the brighter region 214 adjacent to region 215 where there is a sharp drop in light intensity is the chin region. Furthermore, the overall control CPU 101 calculates (extracts) the chin position 207 at the left-right center of region 214 and the position furthest from the neck position 206 (the position shown by the black circle in Figure 8G(a)).

[0239] For example, Figures 8H and 8I show the changes when the face is turned to the right.

[0240] Figure 8H shows the difference image calculated in the same way as in Figure 8E when the user's face is turned to the right. Figure 8I shows the difference image from Figure 8H with the grayscale adjusted to match the light intensity of the reflected infrared light projected onto the user's face and neck, and the double circle indicating the neck position 206, which is the center of neck rotation, and the black circle indicating the chin position 207r superimposed.

[0241] Because the user's face is turned to the right, region 214 moves to region 214r, shown in Figure 8I, which is to the left when viewed from the side of the shooting / detection unit 10. Region 215, which corresponds to the lower part of the face including the lips within face 204, also moves to region 215r, which is to the left when viewed from the side of the shooting / detection unit 10.

[0242] Therefore, the overall control CPU 101 identifies the region 214r in front of 215r, where there is a sharp drop in light intensity, as the chin area. Furthermore, the overall control CPU 101 calculates (extracts) the chin area 207r as the position that is in the left-right center of 214r and is furthest from the neck position 206 (the position shown by the black circle in Figure 8I).

[0243] Subsequently, the overall control CPU 101 determines the movement angle θr, which indicates how far the chin position 207r in Figure 8I has moved, centered on the neck position 206, from the chin position 207 in Figure 8G(a) to the right. As shown in Figure 8I, the movement angle θr is the angle in the left-right direction of the user's face.

[0244] In step S210, the angle of the user's face in the left-right direction is calculated from the chin position detected by the infrared detection processing device 27 of the face direction detection unit 20 (3D detection sensor).

[0245] Next, we will explain how to detect faces facing upwards.

[0246] Figure 10(b) shows the user with their face turned horizontally, and Figure 10(c) shows the user with their face turned 33° above the horizontal.

[0247] In Figure 10(b), the distance from the face direction detection window 13 to the chin tip 203 is denoted as Ffh, and in Figure 10(c), the distance from the face direction detection window 13 to the chin tip 203u is denoted as Ffu.

[0248] As shown in Figure 10(c), the chin tip 203u moves upward along with the face, so it can be seen that the distance between Ffu and Ffh is longer.

[0249] Figure 8J shows the image of the user as seen through the face direction detection window 13 when the user's face is turned 33° upward from the horizontal. As shown in Figure 10(c), the user is looking upward, so the face 204, including the lips and nose, is not visible from the face direction detection window 13 located below the user's chin, and only the tip of the chin 203 is visible. Figure 8K shows the distribution of the light intensity of the reflected light rays 25 when infrared rays 23 are irradiated onto the user in the state shown in Figure 10(c). Figure 8K(a) is a diagram in which the grayscale of the difference image calculated in the same way as in Figure 8E is adjusted to match the light intensity of the reflected light rays of the infrared rays projected onto the user's face and neck, and the symbols of the double circle indicating the neck position 206 and the black circle indicating the chin position 207u are superimposed. The graphs in Figures 8K(b) and (c) show the change in density of the image on the left, with (b) corresponding to the graph in Figure 8F and (c) corresponding to the graph in Figure 8G.

[0250] The six regions 211u to 216u in Figure 8K(a), corresponding to light intensity, are regions with the same light intensity as the region shown in Figure 8F, but with "u" added to them. The light intensity of the user's chin 203 was in the intermediate gray region 214 in Figure 8F(a), but in Figure 8K(a), it has shifted towards the gray side and is in the slightly darker gray region 215u. Thus, as shown in Figure 10(c), the infrared detection and processing device 27 can detect that, as a result of Ffu being at a longer distance than Ffh, the light intensity of the reflected light 25 from the user's chin 203 is weakened inversely proportional to the square of the distance.

[0251] Next, we will explain how to detect faces that are facing downwards.

[0252] Figure 10(d) shows the user with their face tilted 22° downward from the horizontal.

[0253] In Figure 10(d), Ffd is defined as the distance from the face direction detection window 13 to the chin tip 203d.

[0254] As shown in Figure 10(d), since the chin tip 203d moves downward along with the face, the distance of Ffd becomes shorter than that of Ffh, and the light intensity of the reflected light rays 25 from the chin tip 203 becomes stronger.

[0255] Returning to Figure 7C, in step S211, the overall control CPU 101 (distance calculation means) calculates the distance from the chin position to the face direction detection window 13 based on the light intensity of the chin position detected by the infrared detection processing device 27 of the face direction detection unit 20 (3D detection sensor). Based on this, the vertical angle of the face is also calculated.

[0256] In step S212, the overall control CPU 101 stores the angles of the face in the left-right direction (first detection direction) and the vertical direction perpendicular to it (second detection direction), acquired in steps S210 and S211, respectively, in the primary memory 103 as the user's 3D observation direction vi (where i is an arbitrary sign). For example, if the user was observing the center of the front, the observation direction vo would be the vector information [0°,0°], since the left-right direction θh is 0° and the vertical direction θv is 0°. Also, if the user was observing 45° to the right, the observation direction vr would be the vector information [45°,0°].

[0257] In step S211, the vertical angle of the face was calculated by detecting the distance from the face direction detection window 13, but this method is not limited to this method. For example, the angle change may be calculated by comparing the level of variation in the light intensity of the chin tip 203. That is, the angle change of the face may be calculated by comparing the gradient CDh of the reflected light intensity from the base of the chin 202 to the chin tip 203 in Figure 8G(a) with the gradient CDu of the reflected light intensity from the base of the chin 202 to the chin tip 203 in Figure 8K(c).

[0258] Figure 7D is a flowchart of the subroutine for determining the recording direction and range in step S300 of Figure 7A. Before explaining the details of this process, we will first describe the ultra-wide-angle video for which the recording direction and range are determined in this embodiment using Figure 11A.

[0259] In this embodiment, the camera body 1 achieves the acquisition of an image in the observation direction by having the shooting unit 40 capture an ultra-wide-angle image around the shooting / detection unit 10 using an ultra-wide-angle imaging lens 16, and then cropping a portion of that image.

[0260] Figure 11A shows the target field of view 125 in the ultra-wide-angle image captured by the shooting unit 40 when the user is facing forward.

[0261] As shown in Figure 11A, the image-capable pixel area 121 of the solid-state image sensor 42 is a rectangular area. The effective projection area 122 (predetermined area) is the area of ​​a circular hemispherical image projected onto the solid-state image sensor 42 using a fisheye lens 16. The imaging lens 16 is adjusted so that the center of the pixel area 121 and the center of the effective projection area 122 coincide.

[0262] The outermost edge of the circular effective projection area 122 indicates the position of an FOV (Field of View) angle of 180°. When the user is looking at the horizontal and vertical center, the angular range of the target field of view 125, which is the area to be imaged and recorded, is 90° (half of the VOF angle) centered on the center of the effective projection area 122. In this embodiment, the imaging lens 16 can also introduce light rays from outside the effective projection area 122, and can project light rays up to a maximum FOV angle of approximately 192° onto the solid-state image sensor 42 using a fisheye projection. However, beyond the effective projection area 122, the optical performance deteriorates significantly, with extreme drops in resolution, light intensity, and distortion. Therefore, in this embodiment, the recording area will be explained using an example where the image in the observation direction is extracted only from the image projected onto the pixel area 121 of the hemispherical image displayed on the effective projection area 122 (hereinafter simply referred to as the ultra-wide-angle image).

[0263] In this embodiment, the vertical size of the effective projection area 122 is larger than the shorter side size of the pixel area 121, so the images at the upper and lower ends of the effective projection area 122 are outside the pixel area 121, but this is not limited to this. For example, the configuration of the imaging lens 16 may be changed to design the optical system so that the entire effective projection area 122 fits within the pixel area 121.

[0264] The invalid pixel region 123 is the pixel region of the pixel region 121 that was not included in the effective projection region 122.

[0265] The target field of view 125 is the area that extracts the image in the user's viewing direction from the ultra-wide-angle image, and is defined by a preset left, right, up, and down field of view (here, 45°, FOV angle 90°) centered on the viewing direction. In the example in Figure 11A, the user is facing forward, so the center of the target field of view 125 is the viewing direction vo, which is the center of the effective projection area 122.

[0266] The ultra-wide-angle image shown in Figure 11A includes subject A131, which is a child; subject B132, which is a staircase that the child (subject A) is about to climb; and subject C133, which is a toy train shaped like a locomotive.

[0267] Next, Figure 7D shows the recording direction and range determination process in step S300, which is performed to obtain an image of the observation direction from the ultra-wide-angle image described using Figure 11A above. The following explanation of this process will be given using Figures 12A to 12G, which are specific examples of the target field of view 125.

[0268] In step S301, the previously set field of view setting value V is obtained by reading it from the primary memory 103.

[0269] In this embodiment, all field of view angles that allow the image in the observation direction to be extracted from the ultra-wide-angle image by the image extraction / development processing unit 50—45°, 90°, 110°, and 130°—are stored as field of view setting values ​​V in the built-in non-volatile memory 102. Furthermore, in any of steps S103, S106, or S108 in Figure 7B, the field of view setting value V included in the various setting values ​​read from the built-in non-volatile memory 102 is set and stored in the primary memory 103.

[0270] Furthermore, in step S301, the observation direction vi determined in step S212 is set as the recording direction, and the image of the target field of view 125, which is extracted from the ultra-wide-angle image with the acquired field of view setting value V centered on this direction, is saved in the primary memory 103.

[0271] For example, if the field of view setting value V is 90° and the observation direction vo (vector information [0°,0°]) is detected by the face direction detection process (Figure 7C), the target field of view 125 (Figure 11A) is set to have an angle range of 90° in the left / right and up / down directions centered on the center O of the effective projection unit 122. Figure 11B shows the image of the target field of view 125 extracted from the ultra-wide-angle image in Figure 11A. In other words, the overall control CPU 101 (relative position setting means) sets the angle of the face direction detected by the face direction detection unit 20 to the observation direction vi, which is vector information indicating the relative position of the target field of view 125 with respect to the ultra-wide-angle image.

[0272] In this case, when the observation direction is vo, the effect of optical distortion due to the imaging lens 16 can be almost ignored, so the shape of the set target field of view 125 becomes the shape of the target field of view 125o (Figure 12A) after distortion conversion in step S303, which will be described later. Hereafter, the target field of view 125 after distortion conversion in the case of the observation direction vi will be called the target field of view 125i.

[0273] Next, in step S302, the pre-set vibration isolation level is obtained by reading it from the primary memory 103.

[0274] In this embodiment, as described above, the vibration isolation level included in the various setting values ​​read from the built-in non-volatile memory 102 in any of steps S103, S106, or S108 is set and stored in the primary memory 103.

[0275] Furthermore, in step S302, the number of spare pixels Pi for vibration damping is set based on the vibration damping level obtained above.

[0276] In the vibration stabilization process, the image is acquired that tracks the amount of shake of the shooting / detection unit 10, and tracks the image in the opposite direction to the shake direction. For this reason, in this embodiment, a preliminary vibration stabilization area necessary for vibration stabilization is provided around the target field of view 125i.

[0277] In this embodiment, a table holding the value of the number of vibration-damping reserve pixels Pis associated with each vibration-damping level is stored in the built-in non-volatile memory 102. For example, if the vibration-damping level is "medium," a vibration-damping reserve area with a width of 100 pixels, which is the number of vibration-damping reserve pixels Pis read from the table, is set to surround the target field of view.

[0278] Figure 12E shows an example in which a pre-vibration isolation area corresponding to a predetermined vibration isolation level is added around the target field of view 125o shown in Figure 12A. Here, we will explain the case where the vibration isolation level is "medium," that is, the number of pre-vibration isolation pixels Pis is 100 pixels.

[0279] As shown by the dotted lines in Figure 12E, anti-vibration spare pixel frames 126o, each with a width of 100 pixels (Pis) corresponding to the number of spare anti-vibration pixels for each of the target field of view 125o, are set up on the top, bottom, left, and right sides.

[0280] Figures 12A and 12E illustrate the case where the observation direction vi coincides with the center O of the effective projection area 122 (the optical axis center of the imaging lens 16) for the sake of simplicity. On the other hand, when the observation direction vi is in the peripheral area of ​​the effective projection area 122, a conversion is necessary to reduce the effects of optical distortion.

[0281] In step S303, the shape of the target field of view 125 set in step S301 is corrected (distortion converted) considering the observation direction vi and the optical characteristics of the imaging lens 16 to generate the target field of view 125i. Similarly, the number of spare pixels Pi for vibration damping set in step S302 is also corrected considering the observation direction vi and the optical characteristics of the imaging lens 16.

[0282] For example, suppose the field of view setting value V is 90° and the user is observing 45° to the right of the center o. In this case, the observation direction vr (vector information [45°, 0°]) is determined in step S212, and the target field of view 125 is the range of 45° to the left and right and 45° up and down, centered on the observation direction vr. However, considering the optical characteristics of the imaging lens 16, the target field of view 125 is corrected to the target field of view 125r shown in Figure 12B.

[0283] As shown in Figure 12B, the target field of view 125r widens towards the periphery of the effective projection area 122, and the position of the observation direction vr is also slightly inward from the center of the target field of view 125r. This is because, in this embodiment, the imaging lens 16 has an optical design similar to that of a stereoscopic fisheye. Note that if the imaging lens 16 is designed as an equidistant fisheye, equisolid angle fisheye, or orthographic fisheye, this relationship will change, and corrections will be made to the target field of view 125 according to its optical characteristics.

[0284] Figure 12F shows an example in which a pre-vibration isolation area 126r, corresponding to the same vibration isolation level "medium" as the pre-vibration isolation area in Figure 12E, is added around the target field of view 125r shown in Figure 12B.

[0285] In the vibration-damping reserve area 126o (Figure 12E), the width of 100 pixels, which is the vibration-damping reserve pixel count Pis, is set for each of the top, bottom, left, and right sides of the target field of view 125o. In contrast, in the vibration-damping reserve area 126r (Figure 12F), the vibration-damping reserve pixel count Pis is corrected and increases as you move towards the periphery of the effective projection area 122.

[0286] Thus, similar to the shape of the target field of view 125r, the shape of the vibration-damping reserve area necessary for vibration isolation, provided around it, also shows that the amount of correction increases towards the periphery of the effective projection section 122, as shown in the vibration-damping reserve area 126r in Figure 12F. This is because, in this embodiment, the imaging lens 16 has an optical design close to that of a stereoscopic fisheye. Note that if the imaging lens 16 is designed as an equidistant fisheye, equisolid angle fisheye, or orthographic fisheye, the relationship will change, and the vibration-damping reserve area 126r will be corrected according to its optical characteristics.

[0287] The process performed in step S303, which sequentially switches the shape of the target field of view 125 and its vibration-damping reserve area, taking into account the optical characteristics of the imaging lens 16, is a complex process. Therefore, in this embodiment, the process in step S303 is performed using a table stored in the built-in non-volatile memory 102 that holds the shape of the target field of view 125i and its vibration-damping reserve area for each observation direction vi. Depending on the optical design of the imaging lens 16 mentioned above, the calculation formula may be stored in the overall control CPU 101, and the optical distortion value may be calculated using that formula.

[0288] In step S304, the position and size of the video recording frame are calculated.

[0289] As described above, the vibration isolation reserve area 126i necessary for vibration isolation is provided around the target field of view 125i. However, as the position of the observation direction vi approaches the periphery of the effective projection area 122, its shape becomes quite unusual, for example, as seen in the vibration isolation reserve area 126r.

[0290] The overall control CPU 101 can extract and process only the image within this specially shaped area. However, it is not common to use non-rectangular images when recording them as video data in step S600 or transferring them to the display device 800 in step S700. Therefore, in step S304, the position and size of a rectangular video recording frame 127i that encompasses the entire vibration-damping reserve area 126i are calculated.

[0291] Figure 12F shows the video recording frame 127r, indicated by a dashed line, which was calculated in step S304 for the vibration isolation reserve area 126r.

[0292] In step S305, the position and size of the video recording frame 127i calculated in step S304 are recorded in the primary memory 103.

[0293] In this embodiment, the upper left coordinates Xi,Yi of the video recording frame 127i in the ultra-wide-angle image are recorded as the position of the video recording frame 127i, and the width WXi and height WYi of the video recording frame 127i from coordinates Xi,Yi are recorded as the size of the video recording frame 127i. For example, for the video recording frame 127r shown in Figure 12F, the shown coordinates Xr,Yr, width WXr, and height WYr are recorded in step S305. Note that the coordinates Xi,Yi are XY coordinates with a predetermined reference point, specifically the optical center of the imaging lens 16, as the origin.

[0294] Once the vibration-damping reserve area 126i and the video recording frame 127i have been determined in this manner, the subroutine shown in Figure 7D is exited.

[0295] Up to this point, in order to simplify the explanation of the complex optical distortion transformation, we have used observation directions vi that include horizontal 0°, i.e., observation direction vo (vector information [0°,0°]) and observation direction vr (vector information [45°,0°]) as examples of observation direction vi. However, in reality, the user's observation direction vi will be in various directions. Therefore, the following explains the recording range development process that is performed when the horizontal is not 0°.

[0296] For example, when the field of view setting value V is 90° and the observation direction vm[-42°,-40°], the target field of view of 125m is as shown in Figure 12C.

[0297] Furthermore, even with the same observation direction vm (vector information [-42°,-40°]) as the target field of view of 125m, if the field of view setting value V is 45°, the target field of view becomes 128m, which is slightly smaller than the target field of view of 125m, as shown in Figure 12D. In addition, for the target field of view of 128m, a vibration-damping reserve area 129m and a video recording frame 130m are set, as shown in Figure 12G.

[0298] Step S400 is a basic imaging operation and uses a general sequence for the imaging unit 40, so a detailed explanation is omitted. In this embodiment, the imaging signal processing circuit 43 in the imaging unit 40 also performs a process to correct the signal output from the solid-state image sensor 42, which is in a specific output format (examples of standards: MIPI, SLVS), into imaging data using a general sensor readout method.

[0299] Furthermore, if the mode selected by the imaging mode switch 12 is video mode, the shooting unit 40 starts recording when the start switch 14 is pressed. Recording then ends when the stop switch 15 is pressed. On the other hand, if the mode selected by the imaging mode switch 12 is still image mode, the shooting unit 40 takes a still image each time the start switch 14 is pressed.

[0300] Figure 7E is a flowchart of the subroutine for the recording range development process in step S500 of Figure 7A.

[0301] In step S501, the raw data of the entire area of ​​the imaging data (ultra-wide-angle video) generated by the imaging unit 40 in step S400 is acquired and input to the video acquisition unit called the head unit (not shown) of the overall control CPU 101.

[0302] Next, in step S502, based on the coordinates Xi, Yi, width WXi, and height WYi recorded in the primary memory 103 in step S305, the portion of the video recording frame 127i is cut out from the ultra-wide-angle video acquired in step S501. After this cutout, a cropping and development process consisting of steps S503 to S508 (Figure 7F) is started, which is performed only on the pixels within the vibration-damping reserve area 126i. This significantly reduces the amount of computation compared to performing development on the entire area of ​​the ultra-wide-angle video read in step S501, thereby reducing computation time and power consumption.

[0303] Furthermore, as shown in Figure 7F, if the mode selected by the imaging mode switch 12 is video mode, the processes in steps S200 and S300 and the process in step S400 are executed in parallel at the same or different frame rates. In other words, each time the Raw data for the entire area of ​​one frame generated by the imaging unit 40 is acquired, cropping and development processing is performed based on the coordinates Xi, Yi, width WXi, and height WYi recorded in the primary memory 103 at that time.

[0304] When the crop development process for pixels within the vibration-damping reserve area 126i is started, first, in step S503, color interpolation is performed to complete the color pixel information arranged in the Bayer array.

[0305] After that, the white balance is adjusted in step S504, and then the color conversion is performed in step S505.

[0306] In step S506, gamma correction is performed to correct the gradation according to a pre-set gamma correction value.

[0307] In step S507, edge enhancement is performed according to the image size.

[0308] In step S508, the data is converted into a data format that can be temporarily stored by performing compression and other processing, recorded in the primary memory 103, and then the subroutine is exited. Details of this data format that can be temporarily stored will be described later.

[0309] Furthermore, the order and whether or not the cropping and development processes performed in steps S503 to S508 are performed may be adjusted according to the camera system and do not limit the present invention.

[0310] Furthermore, if video mode is selected, the process from steps S200 to S500 will be repeatedly executed until recording is finished.

[0311] This process significantly reduces the amount of computation required compared to processing the entire area read in step S501. As a result, an inexpensive and low-power microcontroller can be used as the overall control CPU 101, and heat generation in the overall control CPU 101 is suppressed, while the battery life of the battery 94 is also improved.

[0312] Furthermore, in this embodiment, in order to reduce the control load on the overall control CPU 101, the optical correction processing (step S800 in Figure 7A) and vibration damping processing (step S900 in Figure 7A) of the image are not performed by the camera body 1, but are transferred to the display device 800 and then performed by the display device control unit 801. Therefore, if only the image data partially extracted from the projected ultra-wide-angle image is sent to the display device 800, the optical correction processing and vibration damping processing cannot be performed. In other words, the extracted image data alone does not contain the position information used to substitute into formulas during optical correction processing or to refer to correction tables during vibration damping processing, so these processes cannot be correctly executed in the display device 800. For this reason, in this embodiment, not only the extracted image data but also correction data including information on the extraction position from the ultra-wide-angle image is sent from the camera body 1 to the display device 800.

[0313] If the extracted image is a still image, the still image data and correction data are transmitted separately to the display device 800, but since there is a one-to-one correspondence between the still image data and the correction data, the display device 800 can correctly perform optical correction processing and image stabilization processing. On the other hand, if the extracted image is a video, transmitting the video data and correction data separately to the display device 800 makes it difficult to determine which correction data corresponds to which frame of the video. In particular, if the clock rate of the overall control CPU 101 in the camera body 1 and the clock rate of the display device control unit 801 in the display device 800 are slightly different, synchronization between the overall control CPU 101 and the display device control unit 801 becomes impossible after several minutes of video capture. As a result, the display device control unit 801 may correct the frame that should be processed with correction data different from the corresponding correction data, leading to problems such as this.

[0314] Therefore, in this embodiment, when transmitting video data extracted from the camera body 1 to the display device 800, correction data is appropriately added to the video data. The method for doing so will be described below.

[0315] Figure 14 is a flowchart of the subroutine for the primary recording process in step S600 of Figure 7A. This process will be explained below with reference to Figure 15 as well. Figure 14 shows the process when the mode selected by the imaging mode switch 12 is video mode. If the selected mode is still image mode, this process starts from step S601 and ends when step S606 is completed.

[0316] In step S601a, the overall control CPU 101 reads an image of one frame from the video footage developed in the recording range development process (Figure 7E) that has not been processed in steps S601 to S606. The overall control CPU 101 (metadata generation means) also generates correction data, which is metadata for the read frame.

[0317] In step S601, the overall control CPU 101 attaches information about the image extraction position of the frame read in step S600 to the correction data. The information attached here is the coordinates Xi,Yi of the video recording frame 127i acquired in step S305. Alternatively, the information attached here may be vector information indicating the observation direction Vi.

[0318] In step S602, the overall control CPU 101 (optical correction value acquisition means) acquires an optical correction value. The optical correction value is the optical distortion value set in step S303. Alternatively, it may be a correction value corresponding to the lens optical characteristics, such as peripheral light intensity correction value or diffraction correction value.

[0319] In step S603, the overall control CPU 101 attaches the optical correction values ​​used for distortion conversion in step S602 to the correction data.

[0320] In step S604, the overall control CPU 101 determines whether or not the camera is in vibration stabilization mode. Specifically, if the pre-set vibration stabilization mode is "medium" or "strong," it determines that the camera is in vibration stabilization mode and proceeds to step S605. On the other hand, if the pre-set vibration stabilization mode is "off," it determines that the camera is not in vibration stabilization mode and proceeds to step S606. The reason for skipping step S605 when the vibration stabilization mode is "off" is that this reduces the amount of calculation data for the overall control CPU 101 and the amount of data transmitted wirelessly, which in turn reduces the power consumption and heat generation of the camera body 1. Although the reduction of data used for vibration stabilization processing has been explained here, it is also possible to reduce the peripheral light intensity correction value and the data regarding the presence or absence of diffraction correction, which are included in the optical correction value obtained in step S602.

[0321] In this embodiment, the vibration isolation mode is pre-set by user operation via the display device 800, but it may also be set as the initial setting of the camera body 1. Furthermore, if the camera system is configured to switch between vibration isolation and non-vibration isolation after data is transmitted to the display device 800, step S604 may be omitted, and the system may proceed directly from step S603 to step S605.

[0322] In step S605, the overall control CPU 101 (movement detection means) attaches the vibration isolation mode acquired in step S302 and the gyro data during video capture, which is linked to the frame read out in step S601a and stored in the primary memory 813, to the correction data.

[0323] In step S606, the overall control CPU 101 updates the video file 1000 (Figure 15) with data encoded from the image data of the frame read in step S600 and the correction data to which various data have been attached in steps S601 to S605. If the first frame of the video footage was read in step S601a, the video file 1000 is generated in step S606.

[0324] In step S607, the overall control CPU 101 determines whether the reading of all frames of the video developed in the recording range development process (Figure 7E) has finished. If it has not finished, it returns to step S601a. ​​If it has finished, it exits this subroutine. The generated video file 1000 is saved in the built-in non-volatile memory 102. In addition to being saved in the primary memory 813 and the built-in non-volatile memory 102 as described above, it may also be saved in the large-capacity non-volatile memory 51. Furthermore, a transfer process (step S700 in Figure 7A) is executed to immediately transfer the generated video file 1000 to the display device 800. After being transferred to the display device 800, the video file 1000 may be saved in the primary memory 813.

[0325] In this embodiment, encoding refers to combining video data and correction data into a single file. However, the video data may be compressed, or the combined video data and correction data may be compressed.

[0326] Figure 15 shows the data structure of video file 1000.

[0327] The video file 1000 consists of a header 1001 and a frame section 1002. The frame section 1002 is composed of a frame dataset, which is a set of images for each frame that makes up the video and their corresponding frame metadata. In other words, the frame section 1002 contains as many frame datasets as there are frames in the video.

[0328] In this embodiment, the frame metadata is information encoded with correction data, including the cropping position (in-video position information), optical correction value, and gyro data, if necessary. However, it is not limited to this. For example, the amount of information in the frame metadata may be changed by adding other information to the frame metadata or deleting information in the frame metadata depending on the imaging mode selected by the imaging mode switch 12.

[0329] Header 1001 records the offset value or starting address to the frame dataset for each frame. Alternatively, header 1001 may store metadata such as the time and size corresponding to the video file 1000.

[0330] Thus, in the primary recording process (Figure 14), a video file 1000 containing each frame of the video developed in the recording range development process (Figure 7E) and its metadata is transferred to the display device 800. Therefore, even if the clock rate of the overall control CPU 101 of the camera body 1 and the clock rate of the display device control unit 801 of the display device 800 are slightly different, the display device control unit 801 can reliably perform the correction process of the video developed by the camera body 1.

[0331] In this embodiment, the optical correction value was included in the frame metadata, but the optical correction value may also be applied to the entire video.

[0332] Figure 16 is a flowchart of the subroutine for the transfer process to the display device 800 in step S700 of Figure 7A. Figure 16 shows the process when the mode selected by the imaging mode switch 12 is video mode. If the selected mode is still image mode, this process starts from step S702.

[0333] In step S701, it is determined whether the recording of video footage by the shooting unit 40 (step S400) has finished or is still in progress. If video footage is being recorded (video capture in progress), the recording range development process for each frame (step S500) and the updating of the video file 1000 in the primary recording process (step S600) (step S606) are performed sequentially. Wireless transfer has a high power load, so performing it in parallel with recording would require a large battery capacity for the battery 94 and separate measures to prevent overheating. Also, from the perspective of computing power, performing wireless transfer in parallel with recording increases the computing load, so it is necessary to prepare a high-spec overall control CPU 101, which also increases the cost. In this embodiment, taking these factors into consideration, the system waits for the video footage recording to finish (YES in step S701) before proceeding to step S702 to establish a connection with the display device 800. However, if the camera system of this embodiment has sufficient power supplied from the battery 94 and no additional heat dissipation measures are required, the camera body 1 may be connected to the display device 800 in advance, such as when it is started up or before recording begins.

[0334] In step S702, a connection to the display device 800 is established via the high-speed wireless unit 72 in order to transfer the video file 1000, which has a large amount of data, to the display device 800. The low-power wireless unit 71 is used for transferring low-resolution video (or video) to the display device 800 for checking the field of view, and for sending and receiving various setting values ​​with the display device 800, but it is not used for transferring the video file 1000 because it takes time to transmit.

[0335] In step S703, the video file 1000 is transferred to the display device 800 via the high-speed wireless unit 72. Once the transfer is complete, the process proceeds to step S704, where the connection with the display device 800 is closed, and then the subroutine is exited.

[0336] Up to this point, we have described the case of transferring a single video file containing images of all frames of a single video. However, for long video videos lasting several minutes, it is also acceptable to use multiple video files divided by time units. If the video file has the data structure shown in Figure 15, even if a single video is transferred to the display device 800 as multiple video files, the display device 800 can correct the video without any timing discrepancies with the correction data.

[0337] Figure 17 is a flowchart of the subroutine for the optical correction process in step S800 of Figure 7A. This process will be explained below with reference to Figure 18. As mentioned above, this process is executed by the display device control unit 801 of the display device 800.

[0338] In step S801, the display device control unit 801 (video file receiving means) first receives the video file 1000 from the camera body 1 that was transferred in the transfer process to the display device 800 (step S700). Subsequently, the display device control unit 801 (first extraction means) obtains the optical correction value extracted from the received video file 1000.

[0339] Next, in step S802, the display device control unit 801 (second extraction means) acquires video (an image of one frame obtained by video capture) from the video file 1000.

[0340] In step S803, the display device control unit 801 (frame image correction means) corrects the optical aberrations of the image acquired in step S802 using the optical correction value acquired in step S801, and saves the corrected image to the primary memory 813. When performing optical correction, if cropping is performed from the image acquired in step S802, the image is cropped and processed in a range narrower than the development range (target field of view 125i) determined in step S303 (cropped development area).

[0341] Figure 18 is a diagram illustrating the process of performing distortion correction in step S803 of Figure 17.

[0342] Figure 18(a) shows the position of the subject 1401 as seen by the user with the naked eye during imaging, and Figure 18(b) shows the image of the subject 1401 projected onto the solid-state image sensor 42.

[0343] Figure 18(c) shows the development region 1402 in the image of Figure 18(b). Here, the development region 1402 is the cropped development region explained earlier.

[0344] Figure 18(d) shows the cropped developed image, where the image of the developed region 1402 has been cropped, and Figure 18(e) shows the cropped developed image from Figure 18(d) after distortion correction. Since cropping is performed during distortion correction of the cropped developed image, the field of view of the image shown in Figure 18(e) is even smaller than that of the cropped developed image shown in Figure 18(d).

[0345] Figure 19 is a flowchart of the vibration isolation subroutine in step S900 of Figure 7A. This process will be explained below with reference to Figure 18(f). As mentioned above, this process is executed by the display device control unit 801 of the display device 800.

[0346] In step S901, the display device control unit 801 obtains the gyro data for the current frame and the previous frame from the frame metadata of the video file 1000, and the amount of shake V calculated for the previous frame in step S902 described below. n-1 Det Obtain the following information. Then, from this information, estimate the amount of deviation V n Pre This is calculated. In this embodiment, the current frame is the frame currently being processed, and the previous frame is the frame immediately preceding the current frame.

[0347] In step S902, the display device control unit 801 obtains detailed information on the amount of blur V from the video file. n Det This is how the amount of blur is determined. Blur detection is performed by calculating how much the feature points of the current frame's image have moved from the previous frame.

[0348] Known methods can be adopted for feature point extraction. For example, a luminance information image obtained by extracting only the luminance information of the frame image can be generated, and an image shifted by 1 to several pixels therefrom can be subtracted from the original image, and pixels whose absolute value is greater than or equal to a threshold can be extracted as feature points. Alternatively, an image obtained by applying a high-pass filter to the luminance information image can be subtracted from the original luminance information image, and the extracted edges can be extracted as feature points.

[0349] The amount of movement is calculated by calculating the difference multiple times while shifting the luminance information images of the current frame and the previous frame by 1 to several pixels each, and calculating the position where the difference at the pixels of the feature points decreases.

[0350] Since multiple feature points are required as described later, it is preferable to divide the images of the current frame and the previous frame into a plurality of blocks and extract feature points for each block. The block division depends on the number of pixels and the aspect ratio of the image, but generally 12 blocks of 4×3 to 54 blocks of 9×6 are preferable. If the number of blocks is small, correction of trapezoidal distortion due to tilting of the imaging unit 40 of the camera body 1 or rotational blur in the optical axis direction cannot be accurately performed. However, if the number of blocks is too large, the size of each block becomes small, and since the feature points are close to each other, errors are included. For this reason, an optimal number of blocks is appropriately selected according to the number of pixels, the ease of finding feature points, the angle of view of the subject, and the like.

[0351] To calculate the amount of movement, it is necessary to shift the luminance information images of the current frame and the previous frame by 1 to several pixels each and perform multiple difference calculations, so the amount of calculation increases. Therefore, the actual amount of movement is the rough blur amount V n Pre and the deviation therefrom (how many pixels it is shifted), so it is possible to significantly reduce the amount of calculation by performing difference calculations only in the vicinity of the rough blur amount.

[0352] Next, in step S903, after performing shake prevention processing using the detailed blur amount V n Det obtained in step S902, this subroutine is exited.

[0353] Furthermore, known methods for vibration isolation include Euclidean transformations that allow rotation and translation, affine transformations that allow these actions, and projective transformations that allow trapezoidal correction.

[0354] While the Euclidean transform can correct movement and rotation in the X and Y axes, it cannot correct blur caused by camera shake in the front-to-back direction or in the pan-tilt direction in the shooting unit 40 of the camera body 1. Therefore, in this embodiment, vibration damping is performed using an affine transform that can also correct magnification and skew. The affine transform, in which the coordinates (x,y) of the reference feature point move to coordinates (x',y'), is expressed by the following equation 100.

[0355]

number

[0356] The affine coefficients of the 3x3 matrix in Equation 100 can be calculated if a shift in at least three feature points is detected. However, if the detected feature points are close to each other or lie on a straight line, the vibration isolation treatment will be inaccurate for areas farther from the feature points or off that line. Therefore, it is preferable to select feature points that are far apart from each other and do not lie on a straight line. Thus, if multiple feature points are detected, close feature points are excluded and the remaining points are normalized using the least squares method.

[0357] Figure 18(f) shows the image after applying the image stabilization process in step S903 to the distortion-corrected image shown in Figure 18(e). Because cropping is performed during the image stabilization process, the field of view of the image shown in Figure 18(f) is smaller than that of the image shown in Figure 18(e).

[0358] By performing this type of image stabilization, it is possible to obtain high-quality images with corrected blur.

[0359] The above describes a series of operations performed by the camera body 1 and the display device 800 included in the camera system of this embodiment.

[0360] After the user turns on the power switch 11 and selects the video mode with the imaging mode switch 12, and simply observes the front without turning their face up, down, left, or right, the face direction detection unit 20 first detects the observation direction vo (vector information [0°,0°]) (Figure 12A). Then, the recording direction / angle determination unit 30 extracts the image of the target field of view 125o shown in Figure 12A (Figure 11B) from the ultra-wide-angle image projected onto the solid-state image sensor 42.

[0361] Subsequently, without the user operating the camera body 1, if, for example, the user starts observing the child (subject A131) in Figure 11A, the face direction detection unit 20 first detects the observation direction vm (vector information [-42°,-40°]) (Figure 11C). Then, the recording direction / angle determination unit 30 extracts the image with a target field of view of 125m (Figure 11C) from the ultra-wide-angle image captured by the shooting unit 40.

[0362] In this way, optical correction and vibration damping processing are performed on the display device 800 in steps S800 and S900 for images cropped into various shapes according to the observation direction. As a result, even if the overall control CPU 101 of the camera body 1 has low specifications, even when cropping an image with significant distortion, such as an image of the target field of view of 125m (Figure 11C), it is possible to obtain an image with distortion and shaking corrected, centered on the child (subject A131), as shown in Figure 11D. In other words, the user can obtain an image captured in their observation direction without touching the camera body 1, other than turning on the power switch 11 and selecting a mode with the imaging mode switch 12.

[0363] The pre-configuration mode will now be explained. As mentioned above, the camera body 1 is a small wearable device, so it does not have any operation switches or setting screens for changing its detailed settings. Therefore, in this embodiment, the detailed settings of the camera body 1 are changed on the setting screen of the display device 800 (Figure 13), which is an external device.

[0364] For example, consider a scenario where you want to capture video with both a 90° and a 45° field of view consecutively. In normal video mode, the field of view is set to 90°, so to perform this type of capture, you would first need to capture video in normal video mode, then stop the video capture, switch the display device 800 to the camera body 1 settings screen, and switch the field of view to 45°. However, performing such operations on the display device 800 during continuous capture is cumbersome, and you might miss capturing the footage you want.

[0365] On the other hand, if the pre-setting mode is set to capture video at a 45° field of view, after capturing video at a 90° field of view, simply sliding the capture mode switch 12 to "Pre" will instantly switch to zoomed-in video capture at a 45° field of view. In other words, the user does not need to interrupt the current capture process and perform the cumbersome operation described above.

[0366] The settings configured in pre-setting mode may include not only changing the field of view, but also the image stabilization level, which can be specified as "Strong," "Medium," or "Off," as well as settings for voice recognition, which are not described in this embodiment.

[0367] For example, in the aforementioned imaging situation, if the user continues to observe the child (subject A131) and switches from video mode to pre-setting mode using the imaging mode switch 12, the field of view setting value V changes from 90° to 45°. In this case, the recording direction / field of view determination unit 30 extracts an image of the target field of view 128m, shown by the dotted frame in Figure 11E, from the ultra-wide-angle image captured by the shooting unit 40.

[0368] Even in pre-setting mode, optical correction processing and image stabilization processing are performed on the display device 800 in steps S800 and S900. This allows the camera body 1's overall control CPU 101 to be low-spec, while still images with corrected distortion and shaking are obtained, zoomed in on the child (subject A131), as shown in Figure 11F. The example shown was changing the field of view setting value V from 90° to 45° in video mode, but the same applies to still image mode. The same also applies when the video field of view setting value V is 90° and the still image field of view setting value V is 45°.

[0369] In this way, the user can obtain a zoomed-in image of their own observation direction simply by switching modes using the imaging mode switch 12 on the camera body 1.

[0370] In this embodiment, the case in which the face direction detection unit 20 and the shooting unit 40 are integrally configured in the camera body 1 has been described. However, this is not limited to the case in which the face direction detection unit 20 is mounted on the user's body other than their head, and the shooting unit 40 is mounted on the user's body. For example, the shooting / detection unit 10 of this embodiment can also be installed on the shoulder or abdomen. However, in the case of the shoulder, if the shooting unit 40 is installed on the right shoulder, the subject on the left side may be obstructed by the head, so it is preferable to install multiple shooting means, including on the left shoulder, to compensate. Also, in the case of the abdomen, a spatial parallax occurs between the shooting unit 40 and the head, so it is desirable to be able to perform a correction calculation of the observation direction to correct for that parallax, as shown in Embodiment 3.

[0371] (Example 2) In Example 2, a method for calibrating individual differences and adjustment differences among users who attach the camera body 1 will be explained in detail using Figures 20 to 23.

[0372] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 2, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0373] Users who wear the camera body 1 have individual differences and adjustment differences, such as their physique, the tilt and angle of the neck area where the camera body 1 is attached, the condition of their clothing, such as the collar, when wearing it, and the remaining adjustment of the band parts 82L and 82R. Therefore, the optical axis center of the imaging lens 16 of the camera body 1 and the field of view center when the user is facing forward (hereinafter referred to as the user's natural state) do not usually coincide. For the user, it is desirable to set the center of the recording area (target field of view 125) in which the image is captured, rather than setting the optical axis center of the imaging lens 16 of the camera body 1 directly as the center of the recording area (target field of view 125).

[0374] Furthermore, there are individual differences not only in the user's natural field of view center, but also in the field of view center when the user turns their head in any direction, including up, down, left, right, and diagonally, as well as in the range of motion of the neck. Therefore, there are individual differences in the relationship between the face direction (observation direction) detected by the face direction detection unit 20 and the center position of the target field of view 125 (hereinafter referred to as the field of view center position) set according to that observation direction. Consequently, calibration work is required to associate the face direction with the field of view center position.

[0375] Ideally, the calibration operation should be performed as part of the preparation process (step S100) shown in Figure 7A. While it is usually assumed that the calibration operation will be performed when the camera body 1 is first started up, it may also be performed when a certain amount of time has elapsed since the last calibration, or when the camera body 1 has shifted position relative to the user since the last calibration. The calibration operation may also be performed when the face direction detection unit 20 can no longer detect the user's face. Furthermore, if the system detects that the user has attached or detached the camera body 1, the calibration operation may be performed when the user reattaches it. In this way, it is desirable that the calibration operation be performed as appropriate at the timing deemed necessary for the proper use of the camera body 1.

[0376] Figure 20 shows the details of the calibrator 850 used in the calibration process according to Example 2. In this example, the case in which the calibrator 850 also functions as a display device 800 will be described.

[0377] The calibrator 850 includes the components of the display device 800 shown in Figure 1D: button A802, display unit 803, in-camera 805, face sensor 806, and angular velocity sensor 807, as well as a positioning index 851 and a calibration button 854. Note that button B804, which was used in Example 1, is not used in this embodiment and can be replaced by the calibration button 854 as described later, and is therefore not shown here.

[0378] Figure 20(a) shows the case where the positioning index 851 is a specific pattern displayed on the display unit 803, while Figure 20(b) shows the case where the external appearance of the calibrator 850 is used for the positioning index 851. In the case of Figure 20(b), the positioning index center 852, which will be described later, is calculated from the information of the external shape of the calibrator 850.

[0379] Furthermore, the positioning indicator is not limited to the examples shown in Figures 20(a) and (b). For example, it may be a separate component from the calibrator 850. Any positioning indicator is acceptable as long as it is easy to measure in size and has a shape that is easy for the user to see. For example, it could be the lens cap of the imaging lens 16 or the charging unit of the camera body 1. In any case, the basic concept of the calibration operation is the same, so the explanation below will mainly use the calibrator 850 shown in Figure 20(a) as an example.

[0380] In this embodiment, the calibrator 850 also functions as a display device 800. The calibrator 850 may be a dedicated device, or it may be a general-purpose smartphone or tablet device.

[0381] The positioning index 851 is an index displayed on the display unit 803 of the calibrator 850, and is a figure that allows for the calculation of the width L851a, the height L851b, and the center 852 of the positioning index. In the calibration process described later, the user directs their face towards the vicinity of the center of the positioning index 851, so it is desirable that the positioning index 851 has a shape that is easy to grasp in the center of the field of view. In Figure 20(a), it is shown as a circle with a black circle in the center of a cross, but it is not limited to this shape. Other shapes such as squares, triangles, star shapes, or even illustrations of characters may be used.

[0382] The positioning index 851 is captured by the imaging unit 40 of the camera body 1. Based on the captured image, the display device control unit 801 (position calculation means and distance calculation means) calculates the distance between the imaging / detection unit 10 and the calibrator 850, and the position coordinates of the positioning index 851 that are visible in the image range. In this embodiment, the calibrator 850, which has the functionality of a display device 800, performs these calculations. However, if the calibrator 850 does not have the functionality of a display device 800, the overall control CPU 101 on the camera body 1 performs these calculations.

[0383] The angular velocity sensor 807 can measure the movement of the calibrator 850. Based on the measurement values ​​from the angular velocity sensor 807, the display device control unit 801 calculates movement information indicating the position and orientation of the calibrator 850, which will be described later.

[0384] The calibration button 854 is a button that the user presses when they face the center of the positioning indicator 851 during the calibration process described later. In Figure 20(a), the calibration button 854 is a touch button displayed on the touch panel display unit 803, but buttons A802 and B804 may also function as calibration buttons.

[0385] Next, the calibration process, which is performed when extracting an image from the ultra-wide-angle image captured by the shooting unit 40 according to the direction of the user's face and then processing that image, will be explained in detail using the flowchart in Figure 21.

[0386] Figure 21 is a flowchart of the calibration process according to this embodiment, which is performed in the camera body 1 (first calibration means) and the calibrator 850.

[0387] For illustrative purposes, in Figure 21, the steps in which the camera body 1 and calibrator 850 receive user input are placed in a frame where the user is the primary operator. Also in Figure 21, the steps executed by the display device control unit 801 of the calibrator 850 in response to the user input are placed in a frame where the calibrator 850 is the primary operator. Similarly, in Figure 21, the steps executed by the overall control CPU 101 of the camera body 1 in response to the user input are placed in a frame where the camera body 1 is the primary operator.

[0388] Specifically, in steps S3104 and S3108 of Figure 21, the camera body 1 is the primary operator, while in steps S3101, S3105, and S3106, the user is the primary operator. Furthermore, in steps S3102, S3103, S3106a, S3107, S3107b, and S3110, the calibrator 850 is the primary operator.

[0389] When this process begins, in step S3101, if the calibrator 850 is not powered on, the user operates button A802 to power on the calibrator 850. Similarly, if the camera body 1 is not powered on, the user switches the power switch 11 to ON to power on the camera body 1. After that, the user establishes a connection between the calibrator 850 and the camera body 1. Once this connection is established, the display unit 801 and the overall control CPU 101 each enter calibration mode.

[0390] In step S3101, the user attaches the camera body 1, adjusts the length of the bands 82L and 82R and the angle of the camera body 1, and positions the camera body 1 in a suitable location so that the shooting / detection unit 10 can capture images.

[0391] In step S3102, the display device control unit 801 (first display means) displays the positioning index 851 on the display unit 803.

[0392] Next, in step S3103, the display control unit 801 instructs the user on the instruction display 855 to specify the position where the calibrator 850 should be held. In this embodiment, five locations are specified in order: the front, upper right, lower right, upper left, and lower left. However, the specified positions are not limited to these, as calibration is possible.

[0393] In step S3104, the overall control CPU 101 activates the imaging unit 40 to enable image acquisition, and also activates the face direction detection unit 20 to enable detection of the user's face direction.

[0394] In step S3105, the user holds the calibrator 850 over the designated position indicated in step S3103.

[0395] Next, in step S3106, the user, while maintaining the position of the calibrator 850 in the designated position, turns their face toward the positioning indicator 851, aligns the center of the user's field of view with the positioning indicator 851, and presses the calibration button 854.

[0396] In step S3106a, the display device control unit 801 (second display means) determines whether the user has seen the positioning index center 852 of the positioning index 851 in the center of their field of view. If it is determined that the user has seen it (YES in S3106a), the display device control unit 801 notifies the user in step S3107 via the instruction display 855 that calibration of the specified position will begin, and also redisplays the calibration button 854. If it is determined that the user has seen it (NO in step S3106a), the user repeats the process from step S3105.

[0397] In step S3107a, when the user presses the calibration button 854, the display device control unit 801 sends a calibration instruction to the camera body 1 in step S3107b.

[0398] In step S3108, the overall control CPU 101 (acquisition / detection means) acquires an ultra-wide-angle image in which the positioning index 851 is captured by the imaging unit 40 in response to a calibration instruction from the calibrator 850, and simultaneously detects the face direction with the face direction detection unit 20. Subsequently, the overall control CPU 101 (generation means) calculates the position coordinate information of the center 852 of the positioning index in the ultra-wide-angle image acquired here, and generates information showing the relationship between the calculated position coordinate information and the face direction detected here.

[0399] The details of the processes in steps S3103 to S3108 will be explained below using Figures 22A to 22F.

[0400] Figures 22A to 22F illustrate the calibration operation in the direction directly in front of the user. The calibration operation aligns the center position of the user's field of view in their natural state with the center position of the target field of view 125 in the image captured by the camera unit 40 of the camera body 1.

[0401] Figure 22A shows the screen displayed on the display unit 803 of the calibrator 850 in step S3103 of Figure 21 during the calibration operation in the direction facing the user.

[0402] As shown in Figure 22A, the display unit 803 of the calibrator 850 displays a positioning indicator 851 and an instruction display 855 that shows where the user should place the positioning indicator 851.

[0403] Instruction 855 is a string of characters that instructs the user to position the positioning indicator 851 at the center of their field of view when facing forward. Note that the instructions displayed as instruction 855 are not limited to text; they may also be provided through other means, such as illustrations, photographs, or videos.

[0404] Alternatively, a typical tutorial approach could be used, where instruction indicator 855 is displayed first, followed by positioning indicator 851.

[0405] Figure 22B shows the user holding the calibrator forward in accordance with the instructions shown in Figure 22A.

[0406] The user holds the calibrator 850 forward according to the instructions shown on the instruction display 855 in Figure 22A (step S3105). The user then positions the calibrator 850 so that the positioning indicator 851 is at the center of the field of view when facing the front of the face, and presses the calibration button 854 (Figure 22A) (step S3106). The determination in step S3106a is made in response to the pressing of the calibration button 854. The specific procedure for this determination method will be described later. If the determination in step S3106a is YES, the display control unit 801 changes the instruction display 855 shown in Figure 22A to a notification "Calibration in the forward direction will begin" in step S3107 and displays the calibration button 854.

[0407] Afterward, the user confirms that the instruction display 855 shown in Figure 22A has changed to a notification that says "Starting forward calibration," and then presses the calibration button 854 (step S3107a).

[0408] In response to the pressing of the calibration button 854, a calibration instruction is sent to the camera body 1 in step S3107b, and the shooting unit 40 acquires the captured image in step S3108.

[0409] Figure 22C is a schematic diagram showing the entire ultra-wide-angle image captured by the imaging lens 16 in the state shown in Figure 22B, and Figure 22D is a schematic diagram showing the image after correcting the aberrations of the ultra-wide-angle image shown in Figure 22C.

[0410] Meanwhile, in step S3108, the face direction detection unit 20 acquires the face direction in response to the user pressing the calibration button 854 in the state shown in Figure 22B.

[0411] Figure 22E is a schematic diagram showing the face direction image recorded by the face direction detection unit 20 in step S3108 of Figure 21 during the calibration operation for the user's front direction.

[0412] As described above using Figures 8G to 8K in Example 1, the face direction detection unit 20 calculates the left-right and up-down angles of the face using the distance and angle between the chin position 207, 207r, 207u, etc. and the neck position 206. However, the distance and angle values ​​between the chin position 207, 207r, 207u, etc. and the neck position 206 are not constant, as are individual differences and adjustment differences, such as those represented by the user's physique, as mentioned above, similar to the image center. Therefore, in this embodiment, the relationship between the chin position and the neck position 206 at the time the calibration button 854 is pressed is defined as the value when the user's field of view is centered on the front. This makes it possible to accurately calculate the user's face direction regardless of individual differences or adjustment differences.

[0413] Returning to Figure 21, in step S3109, the overall control CPU 101 determines whether the preparation for frontal calibration is complete. That is, it determines whether the information necessary for calculating the chin position 207, the neck position 206, and the positioning index center 852 has been acquired.

[0414] If the necessary information has not been obtained at this point, it is determined that the preparation for calibration in the forward direction is not complete (NO in step S3109), and the operations from step S3102 are repeated to obtain the missing information. Note that if the necessary information has not been obtained, it is not necessary to perform all the operations from step S3102; only the operations necessary to obtain the missing information may be repeated.

[0415] Here, the determination in step S3106a is made using the face sensor 806 mounted on the calibrator 850 or the in-camera 805. The specific procedure for this determination method will be explained below using the in-camera 805 as an example to demonstrate the calibration operation for the user's frontal direction. Note that the case where the face sensor 806 is used is omitted because although there is a difference in whether the information is two-dimensional or three-dimensional, the basic concept is the same. However, when the face sensor 806 is used for the determination in step S3106a, the face direction detection unit 20 of the camera body 1 will not perform face detection by emitting infrared 23 to the user while infrared 823 is being emitted from the face sensor 806. This is to prevent the infrared 23 and 823 from interfering with each other.

[0416] First, when the user presses the calibration button 854 in Figure 22A in step S3106, the display device control unit 801 takes an image with the in-camera 805 (face detection means) and acquires an in-camera image 858 (Figure 22F) showing the user. Furthermore, the display device control unit 801 detects the user's face 204, including the front of the neck 201, the tip of the chin 203, and the nose, as well as the position information of the shooting / detection unit 10 (shooting unit 40), from the acquired in-camera image 858.

[0417] Using the positional information detected in the in-camera image 858, the display device control unit 801 (determination means) determines in step S3106a whether the user is looking at the positioning index center 852 of the positioning index 851 in the center of their field of view.

[0418] Furthermore, if this determination determines that the user is looking in a different direction, the display device control unit 801 will display information on the instruction display 855 indicating that correct information could not be obtained. This allows the system to instruct the user to repeat the calibration operation.

[0419] Furthermore, the display control unit 801 may determine, using the in-camera image 858, that the shooting / detection unit 10 is tilted beyond a certain point, or that the face direction detection window 13 is blocked or dirty, or that other conditions prevent proper calibration. In such cases, the display control unit 801 may also display information on the instruction display 855 indicating that correct information cannot be acquired.

[0420] Furthermore, it is possible to obtain the information necessary for parallax correction, which will be described later in Example 5, using the in-camera image 858 acquired in step S3106a and the ultra-wide-angle image acquired in step S3108.

[0421] Specifically, before the positioning index 851 is captured by the imaging unit 40 in step S3108, information on the size of the positioning index 851 (width L851a and height L851b) is transmitted in advance from the calibrator 850 to the camera body 1. This allows the overall control CPU 101 to calculate the distance between the imaging / detection unit 10 and the positioning index 851 using the information on the size of the positioning index 851 and the image of the positioning index 851 captured in the ultra-wide-angle image acquired in step S3108. The positioning index 851 is located in the calibrator 850, which has the same housing as the in-camera 805, and in Figure 22B, the calibrator 850 is facing the user almost directly, so the distance between the in-camera 805 and the imaging / detection unit 10 is equal to the distance between the imaging / detection unit 10 and the positioning index 851.

[0422] Similarly, before the in-camera image shown in Figure 22F is captured by the in-camera 805 in step S3106a, information about the size of the imaging / detection unit 10 is transmitted in advance from the camera body 1 to the calibrator 850. This allows the display device control unit 801 (vertical distance calculation means) to estimate the vertical distance 5070 between the optical axis center of the imaging lens 16 and the user's viewpoint position using the information about the size of the imaging / detection unit 10 and the image of the imaging / detection unit 10 captured in the in-camera image 858 in Figure 22F. In addition, the display device control unit 801 can also estimate the distance 2071 between the imaging lens 16 and the user's chin 203. The distance 2071 may also be the distance between the face direction detection window 13 and the chin 203.

[0423] Here, in order for the face direction detection unit 20 to calculate the user's neck position 206 and chin position, the user's face must be at a certain distance or more from the face direction detection window 13, depending on the design of the face direction detection unit 20. Therefore, this estimation result can be used as one of the criteria for determining whether the face direction detection unit 20 can correctly detect the face direction.

[0424] Returning to Figure 21, if, in step S3109, the overall control CPU 101 determines that it has acquired the necessary information and that the preparation for calibration in the forward direction is complete, it proceeds to step S3110.

[0425] In step S3110, the display unit control unit 801 (first calibration means) calculates the information necessary to offset the cutting center position in order to absorb individual differences and adjustment differences, and offsets the cutting center position based on that information.

[0426] The specific details of the calculation in step S3110 are as follows:

[0427] If the user is in an ideal state according to the design specifications and the camera body 1 is ideally attached, the center 856 of the ultra-wide-angle image acquired in step S3108 shown in Figure 22C and the position of the positioning index center 852 in that ultra-wide-angle image should approximately coincide. However, in reality, due to individual differences and adjustment differences such as the user's physique as described above, the positions of the center 856 and the positioning index center 852 in the ultra-wide-angle image usually do not coincide.

[0428] For the user, the cropping center position should preferably be the center of the field of view in the user's posture and movement, i.e., the position of the positioning index center 852 in the ultra-wide-angle image, rather than the center 856 of the ultra-wide-angle image indicated by the camera body 1.

[0429] Therefore, the amount of displacement between the positioning index center 852 and the center 856 in the ultra-wide-angle image is measured, and the cropping center position is offset to a position based on the positioning index center 852 rather than the center 856 of the ultra-wide-angle image. The face direction detected by the face direction detection unit 20 is also offset in the same manner.

[0430] The specific offset method will be explained with reference to Figures 22C and 22D. As shown in Figure 22C, the amount of displacement of the positioning index center 852 relative to the center 856 of the ultra-wide-angle image is measured, and this is divided into a horizontal displacement 857a and a vertical displacement 857b. After performing an appropriate transformation process according to the projection method for the entire field of view, the offset amount can be determined.

[0431] Alternatively, as shown in Figure 22D, the offset amount may be determined after performing an appropriate transformation process on the ultra-wide-angle image according to the projection method. That is, the amount of displacement between the center 856a and the positioning index center 852 in the transformed ultra-wide-angle image may be measured, and the offset amount may be determined by dividing the displacement into a left-right displacement 857c and a up-down displacement 857d.

[0432] The choice of offset method, as shown in Figure 22C or Figure 22D, can be arbitrarily determined by considering the processing load and purpose of the camera system.

[0433] By performing the forward-facing calibration operation described above, it becomes possible to appropriately correlate the face direction of each user when wearing the device, the center of the field of view in that face direction within the ultra-wide-angle image, and the face direction of the face direction detection unit 20, regardless of individual differences or adjustment differences.

[0434] Up to this point, we have explained the calibration procedure for the front direction, out of the five directions: front, upper right, lower right, upper left, and lower left. However, the same calibration procedure must also be performed for the four directions: upper right, lower right, upper left, and lower left.

[0435] Therefore, in Figure 21, once the processing in step S3110 is completed, the process proceeds to step S3111.

[0436] In step S3111, if it is determined that there is one of the five directions (front, upper right, lower right, upper left, lower left) in which the calibration operation has not yet been performed, the direction in which the calibration operation is performed is changed to that one direction, and the process returns to step S3103. This repeats the calibration operation for the remaining directions, excluding the front direction which has already been completed.

[0437] Although not shown in Figure 21, if it is determined in step S3111 that there are no directions where calibration has not been performed, this process is terminated.

[0438] Figures 23A to 23E illustrate the calibration operation in the direction of the user's right hand upwards (upper right direction in ultra-wide-angle images). Figures 23A to 23E correspond to Figures 22A to 22E, respectively, and the basic operation is the same, so common explanations are omitted.

[0439] Here, as shown in Figure 23A, the instruction display 855 shows textual instructions to position the positioning indicator 851 at the center of the field of view when the face is turned to the upper right.

[0440] Figure 23B shows the user holding the calibrator 850 to the upper right in accordance with the instructions shown on the instruction display 855 in Figure 23A.

[0441] Figure 23C is a schematic diagram showing the entire ultra-wide-angle image captured by the imaging lens 16 in the state shown in Figure 23B.

[0442] As shown in Figure 23C, the specific offset method involves first measuring the displacement between the center 856 and the positioning index center 852 in the ultra-wide-angle image. Then, the measured displacement is divided into a diametrical displacement 857e and an angular displacement 857f, and the offset amount is determined after performing an appropriate transformation process according to the projection method for the entire field of view.

[0443] Alternatively, as shown in Figure 23D, the offset amount may be determined after performing an appropriate transformation process on the ultra-wide-angle image according to the projection method. That is, the amount of displacement between the center 856a and the positioning index center 852 in the transformed ultra-wide-angle image may be measured, and the offset amount may be determined by dividing the displacement into a diametrical displacement 857g and an angular displacement 857h.

[0444] In the method of determining the offset amount explained using Figures 22A to 22E, the displacement was divided into vertical and horizontal directions. In contrast, in the method of determining the offset amount explained using Figures 23A to 23D, the displacement was divided into diametrical and angular directions. However, this difference in method is merely for the sake of explanation, and either method may be used.

[0445] Furthermore, as shown in Figure 23E, the face direction detection unit 20 is able to acquire the neck position 206 and chin position 207ru, which are necessary for calculating the face direction when the user is facing upwards and to the right. Therefore, regardless of individual differences or adjustment differences among users, the face direction when the user is looking in the direction of the positioning index center 852 (upper right in this case) can be accurately measured.

[0446] As described above, in the calibration process shown in Figure 21, calibration operations are performed not only in the front direction but also in the upper right, lower right, upper left, and lower left directions. This allows the face direction detection unit 20 to correctly measure which direction the user is facing when the user turns their head in any of the up, down, left, or right directions, enabling the camera body 1 to be used appropriately regardless of individual differences or adjustment differences.

[0447] In the above description, for simplicity, we explicitly explained how to perform repeated calibration operations in five directions: front, upper right, lower right, upper left, and lower left.

[0448] However, the calibration operation is not limited to this method. For example, the user may continuously move the calibrator 850 along a trajectory such as a Z-shape, spiral, or polygonal shape according to the instruction display 855, while simultaneously keeping the positioning index 851 displayed on the calibrator 850 in the center of the field of view. In this method, the display device control unit 801 sends calibration instructions to the camera body 1 multiple times while the calibrator 850 is moving in this manner. Each time the overall control CPU 101 receives a calibration instruction, it acquires the face direction detected by the face direction detection unit 20 and the position coordinate information of the center of the positioning index 852 in the ultra-wide-angle image captured by the shooting unit 40, and stores this as history information. Subsequently, the overall control CPU 101 combines the information extracted from the acquired history information to calculate the relationship between the center position of the image crop and the face direction of the user. Furthermore, by using the information from the in-camera 805 and face sensor 806 acquired by the calibrator 850 while the calibrator 850 is moving using this method, the information extracted from the history information may be limited to information about the state in which the user is looking at the positioning index 851. This prevents information about the state in which the user is looking away from the history information, for example, and thus improves the accuracy of relationship calculations.

[0449] Furthermore, the display device control unit 801 may also transmit the measurement value from the angular velocity sensor 807 to the camera body 1 when a calibration command is issued. In this case, the overall control CPU 101 acquires movement information from the transmitted measurement value from the angular velocity sensor 807, indicating how the user moves the calibrator 850 and the position and orientation of the calibrator 850, and also stores this as history information. This makes it possible to perform the calibration operation simply and accurately using the movement information based on the measurement value from the angular velocity sensor 807, the face direction detected by the face direction detection unit 20, and the position coordinate information of the positioning index center 852 in the ultra-wide-angle image captured by the shooting unit 40.

[0450] However, in this case, the movement information based on the measurement value from the angular velocity sensor 807 and the movement information based on the position coordinate information from the positioning index 851 must match. Therefore, when using the measurement value from the angular velocity sensor 807, it is necessary to synchronize the communication between the camera body 1 and the calibrator 850.

[0451] In Example 2, a calibration method was described that relates the user's face direction to the center position of the target field of view 125 in the ultra-wide-angle image, even with individual differences and adjustment differences. However, the present invention is not limited to the various forms exemplified in Example 2, and various modifications and changes are possible within the scope of its gist.

[0452] (Example 3) In Example 3, a method for preventing motion sickness caused by secondarily recorded video will be explained using Figures 24-26.

[0453] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 3, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0454] Thanks to advancements in video technology, it's now easy to enjoy CG that's indistinguishable from live-action footage and impressive 3D images.

[0455] On the other hand, when such 3D images are dynamic or shaky, such as those found in VR, viewers are more likely to experience motion sickness. Motion sickness can cause symptoms similar to motion sickness, and there is growing interest in safety measures to address this.

[0456] Assume the camera system is designed to extract and develop the image in the direction the user's face is facing during the recording range development process (step S500). In this case, if the user's face moves quickly during imaging by the shooting unit 40 (step S400), the video scene will also switch at a rapid pace.

[0457] While the user themselves may not experience motion sickness due to rapid facial movements during imaging by the camera unit 40, if the resulting video, which was secondarily recorded in step S1000, includes such scenes, viewers watching the video may experience motion sickness.

[0458] Patent documents 1 and 2, which visualize the direction the face is facing, do not contain any descriptions of countermeasures against this type of motion sickness caused by video.

[0459] Therefore, in this embodiment, even when the user moves their face quickly when the shooting unit 40 is capturing images, the resulting image is controlled so that it does not contain fast-changing video scenes, thereby providing a camera system that prevents viewers from experiencing motion sickness.

[0460] As explained using Figures 8H to 8K and Figures 10(b) to (d), the user's face rotates in the up, down, left, and right directions when the imaging unit 40 takes an image.

[0461] Therefore, below, the direction and speed of movement of the user's face will be expressed by angular velocity ω, and the amount of movement will be expressed by angle Θ.

[0462] The angular velocity ω is calculated by dividing the angle Θ detected by the face direction detection unit 20 by the detection interval.

[0463] Here, examples of actions in which people quickly move their heads include turning around, glancing, and observing moving objects.

[0464] Turning around is the action of quickly turning your head, for example, when a loud noise occurs behind you.

[0465] A quick glance is the action of noticing something in your field of vision, looking at it, but then returning your face to its original position because you're not particularly interested.

[0466] Observing moving objects involves actions such as observing birds or kites flying freely across the sky.

[0467] If these actions occur during imaging by the camera unit 40, and the image in the direction the user's face is facing is simply extracted and developed using the recording range development process, as described above, viewers of the resulting image may experience motion sickness.

[0468] Therefore, the overall control CPU 101 determines that if an angular velocity ω that is greater than or equal to the threshold ω0 is calculated for a predetermined time or longer (a first predetermined time or longer), the user has performed an action of quickly moving their face (turning around, glancing, or observing a moving object). Furthermore, if the overall control CPU 101 determines that the action that occurred is not a glancing or observing a moving object using the method described later with reference to Figure 25, it determines that the action is turning around. In this case, the overall control CPU 101 does not extract the image in the direction the user's face is facing directly using the recording range development process, but rather performs delayed extraction, which extracts the image with a delay relative to the movement of the user's face.

[0469] In this embodiment, the threshold ω0 is set to π / 8 rad / s. This is the speed at which a face moves from 0 degrees forward to 90 degrees to the side in 4 seconds. However, the threshold ω0 is not limited to π / 8 rad / s. For example, based on the frame rate n fps, the threshold ω0 may be set to (n × π) / x rad / s (where x is any value).

[0470] The angular velocity ω is the angle Θ obtained from the image of frame n in this case.n and the time of acquisition t n And the angle Θ obtained from the image of the previous frame n-1. n-1 and the time of acquisition t n-1 Therefore, angular velocity ω n This can be calculated using the following formula. ω n =(Θ n -Θ n-1 ) / (t n -t n-1 ) However, the angular velocity ω is the angular velocity ω of the nx frame x frames prior. n-x From this, the angular velocity ω of frame n n Alternatively, it could be calculated as the average of the x frames up to that point.

[0471] Furthermore, in this embodiment, the predetermined time is set to 0.2 seconds, but this value is not the only one that is specified.

[0472] The following explanation of delayed data extraction when the user is reviewing data will be used with reference to Figure 24.

[0473] In the explanation of Figures 11 and 12 in Example 1, the distortion of the imaging lens 16 was taken into consideration, but in this example, for the sake of simplicity, the distortion of the imaging lens 16 is not considered. Furthermore, the calibration process in Example 2 is performed on the frame image, and the center of the image in each frame is assumed to coincide with the center of the user's field of view at the time the image is captured. In addition, to explain the case where the face is turned directly to the side, an example is given in which light rays up to a maximum FOV angle of approximately 192° are projected onto the solid-state image sensor 42.

[0474] Region 4000 represents the image-capable pixel area of ​​the solid-state image sensor 42.

[0475] Image 4001 (Figures 24(a) and 24(b)) shows frame f, cropped with a field of view of 125, aiming at the direction the face is currently facing. n This is an image.

[0476] Image 4002 (Figures 24(a) and 24(b)) is a frame f cropped with a field of view of 125, aiming at the direction the face was facing last time. n-1 This is an image.

[0477] Below, frame f n-1 From the center of image 4002, frame f n Let d be the value of the distance 4010 (Figure 24(a)) to the center of image 4001.

[0478] Image 4003 (Figure 24(b)) shows the delayed extracted frame f' when the angular velocity ω of the face, based on the face direction detected by the face direction detection unit 20, is greater than or equal to the threshold ω0. n This image is extracted from the video projected onto area 4000.

[0479] Below, frame f n-1 Delayed crop frame f' from the center of image 4002 n Let d' be the value of the distance 4011 to the center of image 4003.

[0480] The delay distance of 4012 corresponds to frame f n Frame f' from the center of image 4001 n This is the distance to the center of image 4003, and we will refer to this value as "d" below.

[0481] In this case, the value d for distance 4010 is greater than the value d' for distance 4011 (d>d').

[0482] Next, we will explain an example of how to determine the value of d' using Figure 24(c).

[0483] This section describes the case where the user quickly moves their face 90 degrees to the right from the front (observation direction vo (vector information [0°,0°])). In this case, the frame f where the face is facing forward (observation direction vo (vector information [0°,0°])) n After image 4021 is obtained, a short time later, frame f shows the face turned 90° to the right. n+x Image 4022 is obtained.

[0484] To prevent motion sickness from video, if the viewer needs to move 90° to the right from the front over a period of t seconds (e.g., 4 seconds) or more, then if the video's frame rate is n fps (e.g., 30 fps), then d'=(f n+x -f n ) / (n fps × t seconds).

[0485] Meanwhile, frame f n From frame f' n As the distance d'' to the point increases, the direction the face is facing is not recorded as the recording direction, so frame f n This means that the subject the user was looking at may not be visible in the image.

[0486] Therefore, the delay time is a predetermined time Th delay If the time exceeds (the second predetermined time), the delayed segmentation will be stopped, and segmentation will be performed in the direction the face is currently facing.

[0487] Here, the delay time refers to the time t0 when the delay began (step S4211 in Figure 26) and the current time t when the face continues to move. n This is the difference in step S4213 (in Figure 26).

[0488] Predetermined value Th delay In this embodiment, it is set to 1 second, but is not limited to this. For example, based on the frame rate n fps, a predetermined value Th delay You may set this to 20 / n seconds. The predetermined value Th delay If the frame rate is 20 / n seconds, the predetermined value Th increases as the frame rate increases. delay This reduces the delay. This is because a higher frame rate reduces the likelihood of motion sickness, allowing the camera to return to the current direction of the face after a short delay.

[0489] On the other hand, if you stop the delayed cut and return to cutting in the direction the face is currently facing, the video scene will switch abruptly. Since such abrupt transitions in video scenes can feel unnatural to the user, you may incorporate fade-outs, fade-ins, or other video effects.

[0490] Additionally, the trajectory of the direction the face is currently facing is saved so that the cropping process can be resumed for that direction.

[0491] Here, we will explain the trajectory of the direction the held face is facing, using Figure 25(a) as an example of a case where it is determined that the user is glancing at the device.

[0492] As mentioned earlier, the process of stopping the delayed segmentation and returning to segmentation in the direction the face is currently facing occurs when the delay time is a predetermined value Th delay This will also be executed if the above conditions are met, but it will also be executed if the person is glancing around, that is, if their face looks in a specific direction and then immediately turns back to its original direction.

[0493] Figure 25(a) shows an example of the trajectory of the direction the user's face is pointing when they are glancing at something.

[0494] Frame f n-3 The center of the image, position 4101, coincides with the user's field of view center when the face begins to move. Subsequently, the user's field of view center is at frame f n-2 ,f n-1 ,f n The image moves to positions 4102, 4103, and 4104, which are the centers of each respective image. Hereafter, this movement of the user's field of view center will be referred to as the face movement vector.

[0495] The user's field of view then remained at position 4104 for a while before moving to frame f nx+1 ,f nx+2 ,f nx+3 It moves to the center positions of each image, 4105, 4106, and 4107, and frames f nx+3 The image stops at position 4107, which is the center of the image.

[0496] In other words, the facial motion vectors at positions 4101-4104 and those at positions 4104-4107 are in opposite directions.

[0497] The overall control CPU 101, when it detects a group of frames in which the facial movement vectors coincide in opposite directions, as illustrated in Figure 25(a), determines that the group of frames is a group of frames in which the user is glancing at the screen.

[0498] In this case, the overall control CPU 101 performs a delayed extraction from the position 4101 where the face began to move until the position 4104 where the face's motion vector began to move in the opposite direction. This is because position 4104 is considered to be the position where the subject the user wanted to glance at is visible.

[0499] On the other hand, after delay extraction up to position 4104, the overall control CPU 101 resumes extraction in the direction the face is currently facing up to position 4107 where the face movement has stopped.

[0500] Furthermore, when the user's face is moving, the overall control CPU 101 detects an object located near the center of the field of view in the direction the face is facing. If the detected object remains in the center of the field of view in the direction the face is facing, the CPU determines that the user is observing a moving object. In this case, no delayed extraction is performed in this embodiment.

[0501] Figure 25(b) shows an example of images for each frame when the user is observing moving objects.

[0502] Frame f n-1 The center of image 4121 coincides with the user's field of view center when the face begins to move. Subsequently, the user's field of view center is at frame f n ,f n+1 ,f n+2 ,f n+3 ,f n+4 It moves to the center position of each of the images 4122-4126.

[0503] The same subject, a bird, remains present near the center of each frame, from image 4121 to 4126.

[0504] When the overall control CPU 101 detects a series of consecutive frames in which the same subject is located near the center of the image, as illustrated in Figure 25(b), it determines that the series of frames is a series of frames in which motion is being observed.

[0505] In this case, the overall control CPU 101 does not perform delayed extraction. This is because performing delayed extraction during motion observation would increase the likelihood that the subject would not be visible in the video.

[0506] Furthermore, if viewers were to view the video, which is extracted from images 4121-4126 in response to the user's rapid facial movements during motion observation, it could potentially cause motion sickness. Therefore, the overall control CPU 101 does not extract images from the frames during motion observation, but instead records the entire pixel area that can be captured by the solid-state image sensor 42, i.e., the entire 4000-area image.

[0507] Furthermore, the threshold ω0, predetermined time, and predetermined value Th mentioned above are also included. delay It may also have a range called a dead zone.

[0508] Next, the motion sickness prevention process according to this embodiment will be explained using the flowchart in Figure 26. This process is executed each time a frame is captured in step S400 while the shooting unit 40 is capturing video.

[0509] In step S4201, the overall control CPU 101 acquires the face direction (observation direction) recorded in the primary memory 103 during the face direction detection process performed for capturing this frame.

[0510] In step S4202, the overall control CPU 101 obtains the position and size (cropping range) of the video recording frame recorded in the primary memory 103 during the recording direction and range determination process performed for this frame capture.

[0511] In step S4203, the overall control CPU 101 (calculation means) calculates the angular velocity ω of the face based on the face direction acquired in step S4201 during the current frame capture, the face direction from the previous frame capture held in the primary memory 103, and the frame rate. Subsequently, the overall control CPU 101 determines whether the face has started moving at an angular velocity ω greater than or equal to the threshold ω0. Specifically, it is determined that the face has started moving at an angular velocity ω greater than or equal to the threshold ω0 if the user's face has moved at an angular velocity ω greater than or equal to the threshold ω0 for a predetermined time (0.2 seconds) or longer. If it is determined that the face has started moving, the process proceeds to step S4204; otherwise, it returns to step S4201. In other words, even if the user's face has moved at an angular velocity ω greater than or equal to the threshold ω0, if the time is less than the predetermined time (less than the first predetermined time), the process returns to step S4201. Furthermore, if the face orientation from the previous frame is not stored in the primary memory 103, and the angular velocity of the face cannot be calculated in step S4203, the process returns to step S4201.

[0512] In step S4204, the overall control CPU 101 determines whether the face has moved by a predetermined angle or more, based on the angular velocity ω of the face calculated in step S4203. If it determines that the face has moved, the process proceeds to step S4206; otherwise, the process proceeds to step S4205. In addition, in step S4204, the overall control CPU 101 may also determine whether the face has moved at a predetermined angular velocity or more for a predetermined time (0.2 seconds) or longer.

[0513] In step S4205, the overall control CPU 101 determines whether the movement of the face has stopped based on the angular velocity ω of the face calculated in step S4203. If it is determined that the movement has stopped, the process returns to step S4201; otherwise, the process returns to step S4204.

[0514] In step S4206, the overall control CPU 101 determines whether the subject being imaged is moving (i.e., whether the user is observing a moving object). If it is determined that the subject is moving, the process proceeds to step S4207; otherwise, the process proceeds to step S4208.

[0515] In step S4207, the overall control CPU 101 decides that in the current frame's recording range development process, it will not perform crop development, but will instead perform development processing on the entire RAW data acquired from the entire solid-state image sensor 42, and proceeds to step S4205.

[0516] In step S4208, the overall control CPU 101 stores the face orientation acquired in step S4201 during the current frame capture in the primary memory 103, and proceeds to step S4209.

[0517] In step S4209, the overall control CPU 101 (delay means) decides that in the recording range development process for the current frame, it will perform crop development (delayed cropping) on ​​a cropping range centered at a position shifted by a distance d from the face direction of the previous frame. Then, the process proceeds to step S4210.

[0518] In step S4210, the overall control CPU 101 determines whether the start time t0 of the delay time stored in the primary memory 103 has been cleared. If it is determined that it has been cleared, the process proceeds to step S4211; otherwise, the process proceeds to step S4212.

[0519] In step S4211, the overall control CPU 101 stores the current time in the primary memory 103 as the start time t0, and then proceeds to step S4212.

[0520] In step S4212, the overall control CPU 101 determines the delay time based on the angular velocity ω of the face calculated in step S4203, and sets the delay time to a predetermined value Th delay Before reaching the next step, it is determined whether the facial movement has stopped. If it is determined that it has stopped, the process proceeds to step S4215; otherwise, the process proceeds to step S4213.

[0521] In step S4213, the overall control CPU 101 stores the current time as time tn in the primary memory 103 and proceeds to step S4214.

[0522] In step S4214, the overall control CPU 101 calculates the delay time, which is the difference between the time tn stored in the primary memory 103 and the start time t0, and the delay time is set to a predetermined time Th delay Determine whether or not the above is true. A predetermined time Th delay If the above is true, proceed to step S4215; otherwise, return to step S4206.

[0523] In step S4215, the overall control CPU 101 clears the start time t0 stored in the primary memory 103 and proceeds to step S4216.

[0524] In step S4216, the overall control CPU 101 determines the recording direction and field of view in the recording direction / field of view determination unit 30 based on the face direction detected by the face direction detection unit 20, and then proceeds to step S4217.

[0525] In step S4217, the overall control CPU 101 records a flag in the metadata for the current frame and returns to step S4201. The metadata flag set here is used to determine the timing for applying video effects such as fade-in and fade-out (fade effects) during the secondary recording process described in step S1000 of Example 1.

[0526] In this embodiment, when the angular velocity ω of the face exceeds the threshold ω0, instead of simply extracting the frame in the direction the face is facing, the frame is extracted according to the movement of the face, which has the effect of reducing motion sickness.

[0527] (Example 4) In Example 4, a method for correcting the video cropping range according to the speed of movement of the user's face direction will be explained using Figures 27 and 28.

[0528] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 4, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0529] First, let's explain how a person changes their direction of observation. Typically, when a person finds something of interest at the edge of their field of vision, away from the center, and turns their direction of observation towards it, their face moves first, and beyond a certain point, their body follows with a delay.

[0530] In other words, in such cases, the direction of the imaging lens 16 on the imaging / detection unit 10 (Figure 10(a)) located in front of the clavicle does not move when only the face is initially turned. Subsequently, when the user begins to turn their entire body, the direction of the imaging lens 16 on the camera body 1 also moves. The following explanation will take these characteristics of human body movement into consideration.

[0531] Furthermore, when the face direction detection unit 20 detects the face direction, variations occur due to detection errors. If the video cropping position is calculated based on the face direction detection result which includes such variations, the video recorded secondarily in step S1000 will have a blur similar to camera shake in a typical video, resulting in a poor appearance. Therefore, in order to correct for the fine detection fluctuations, a low-pass filter is applied to the face direction detection result to remove the fine variations.

[0532] Furthermore, if the face direction is detected by tracking even momentary facial movements, such as checking left and right while walking on a public road, the video recorded secondarily in step S1000 will be prone to causing motion sickness. Therefore, in this embodiment, processing is performed to remove (smooth out) even minute facial movement components detected by tracking momentary facial movements of about 1 to 2 seconds. This makes the video recorded secondarily in step S1000 a more visually appealing video.

[0533] Next, an overview of the cropping range correction process in this embodiment will be explained using Figure 27.

[0534] In the graphs shown in Figure 27, the horizontal axis represents time, and the vertical axis represents the angles of the actual observation center (Figure 27(a)), face direction (Figures 27(b), (c)), direction of the imaging lens 16 (Figure 27(d)), and cropping position (Figures 27(e), (f)), respectively. Note that the upward direction on the vertical axis indicates the rightward direction.

[0535] Figure 27(a) is a graph showing the actual movement of the observation center (face direction). Note that the angle on the vertical axis in Figure 27(a) does not represent the angle indicating the face direction detected by the face direction detection unit 20, but rather the position of the user's face relative to a fixed position such as the ground (ground reference). In other words, the graph in Figure 27(a) shows that the user is initially facing forward, but begins to turn to the right around 1 second.

[0536] Figure 27(b) is a graph showing the detection result (observation direction vi) of the face direction detection unit 20. The line showing the detection result in Figure 27(b) is not smooth because, as mentioned above, there is variability in the detection result due to detection error. Therefore, in this embodiment, a low-pass filter is applied to the detection result of the face direction detection unit 20.

[0537] Furthermore, although not detected in Figure 27(b), the system also removes (smooths out) changes in face direction that are detected by tracking momentary face movements.

[0538] Figure 27(c) is a graph showing the result of applying a low-pass filter to the detection result of the face direction detection unit 20 in Figure 27(b) and smoothing it. As shown in Figure 27(c), the line showing the detection result in Figure 27(b) becomes a smooth line when a low-pass filter is applied. However, by applying such a filter, in Figure 27(c), the movement of the detected face direction from the front to the right begins at around 2 seconds, which is a delay (time lag) compared to Figure 27(b), where the movement begins almost simultaneously with the case in Figure 27(a). Note that the angle on the vertical axis in Figures 27(b) and (c) is the angle from the direction of the imaging lens 16 (camera body 1 reference), unlike the ground reference in Figure 27(a).

[0539] Furthermore, in Figure 27(b), the tilt becomes gentler from around 4 seconds compared to Figure 27(a). This means that, as shown in Figure 27(d), the camera body 1 (direction of the imaging lens 16) began to move together with the user's body from around 4 seconds, and therefore the relative speed of movement in the face direction detected by the face direction detection unit 20 slowed down.

[0540] As shown in Figure 27(e), one possible method is to calculate the cropping position, that is, the observation direction that is the center of the target field of view 125, by adding the amount of movement of the camera body (Figure 27(d)) ​​to the face direction detection result that has been smoothed by applying a low-pass filter (Figure 27(c)). However, if the cropping position is calculated using this simple addition method, the cropping position does not follow the actual movement of the observation center, and the resulting video from the secondary recording appears as if the panning has suddenly accelerated from around 4.5 seconds when the body movement begins.

[0541] In other words, to eliminate the feeling of incongruity with the actual movement of the observation center, it is preferable to calculate the cropping position (expected value) so that the panning is approximately constant, as shown in Figure 27(f).

[0542] Therefore, in this embodiment, the cropping position is calculated in such a way that the panning does not appear to accelerate suddenly, as shown in Figure 27(e). Note that if the movement speed of the cropping position is 0° / sec and 10° / sec, as shown in Figure 27, the expected value in Figure 27(f) can be calculated by adding the amount of movement of the camera body 1 in Figure 27(d), which is 1 second earlier in this embodiment, to the face direction detection result in Figure 27(c) by the aforementioned time lag. However, in reality, the movement speed of the cropping position is only 2 types, meaning that the observation direction does not suddenly accelerate or stop, but decelerates slowly. However, the above calculation method does not allow the expected value to draw a slow deceleration curve. Therefore, in this embodiment, when the movement of the camera body 1 stops, the movement speed of the cropping position from the start to the stop of the movement in the observation direction, or for a certain period in the past, is allocated to several frames so that the expected value draws a deceleration curve.

[0543] The following describes the extraction range adjustment process in this embodiment step by step, using the flowchart in Figure 28.

[0544] In the following explanation, we will simplify or omit explanations of parts that are common to Examples 1-3 described above.

[0545] Figure 28(a) is a flowchart of the subroutine for determining the recording direction and range in step S300 of Figure 7A according to this embodiment.

[0546] In step S4000a, a low-pass filter is applied to the observation direction vi obtained in the face direction detection process of step S200 to smooth it (smoothing means). As described above using Figure 27(b), the observation direction vi has some variation due to detection errors. A simple method for the low-pass filter is to take a simple moving average of the past several times, for example, 5 to 10 times. However, in this case, the more times the average is taken, the slower the tracking becomes when the direction of the face moves. Also, in cases where the user turns to the right and then immediately turns to the left, there is a problem in that the observation direction vi when the user is turned furthest to the right cannot be detected.

[0547] Furthermore, since the degree of detection error varies depending on the detection method, it is preferable to adjust the degree of smoothing as appropriate depending on the detection method. It is also possible to apply the low-pass filter differently in the vertical and horizontal directions.

[0548] As mentioned above, momentary facial movements often do not need to be recorded in order to preserve the user's experience as video. For example, this includes situations where the user has to move their head to check for safety on the left and right while walking. Images captured at such moments do not need to be recorded. Therefore, in this embodiment, the observation direction vi acquired when the user returns to approximately the original direction in about 2 seconds is also smoothed in step S4000a.

[0549] Furthermore, while safety checks are often necessary in the left-right and downward directions, they are less necessary in the upward direction, so it may be acceptable to omit the application of a low-pass filter in the upward direction.

[0550] Once the cutting range is determined by the processes from steps S301 to S304 (Figure 7D), the process proceeds to step S4000, where the overall control CPU 101 (second calibration means) executes the cutting range correction process.

[0551] After that, the corrected cropping range is recorded in step S305, and then this subroutine is exited. The cropping range correction process will be explained using the flowchart in Figure 28(b).

[0552] Figure 28(b) is a flowchart of the cropping range correction process in step S4000.

[0553] In Figure 28(b), first, in step S4001, the overall control CPU 101 (movement speed calculation means) acquires gyro information from the angular velocity sensor 107, that is, the movement of the camera body 1 in the current frame (gyro movement amount).

[0554] In this embodiment, an angular velocity sensor 107 was used, but the method is not limited to this as long as the movement of the camera body 1 can be detected. For example, a magnetic sensor that measures the magnitude and direction of a magnetic field (not shown) may be used, or an acceleration sensor 108 that detects acceleration may be used. Furthermore, a method may be used in which a movement vector is detected by extracting feature points and calculating how much those feature points have moved, and the amount of movement of the camera body 1 is calculated. Known methods can be used for extracting feature points. For example, a bandpass filter can be applied to an image from which only the brightness information of two images has been extracted to extract edges, and the amount of movement can be calculated by subtracting multiple edge images with a shift in position and calculating the position where the difference is smallest. This method increases the computational load, but it is one of the preferred methods because it eliminates the need for hardware such as the angular velocity sensor 107, thus allowing for a lighter camera body 1.

[0555] The following explanation continues using the example of acquiring gyro information from the angular velocity sensor 107.

[0556] In step S4002, the movement speed of the camera body 1 (gyro movement speed) is calculated from the gyro information acquired in step S4001 and previously acquired gyro information.

[0557] In step S4003, it is determined whether the gyro movement speed calculated in step S4002 is decreasing. If the movement speed is not decreasing (NO in step S4003), proceed to step S4004; otherwise, proceed to step S4006.

[0558] In step S4004, the overall control CPU 101 (second calibration means / observation direction correction means) calculates the movement speed of the cropping position from the cropping position determined in step S304 and the cropping position acquired in the past. Next, the overall control CPU 101 obtains a subtraction amount obtained by subtracting the gyro movement speed acquired earlier by the time lag caused by applying a low-pass filter from the calculated movement speed of the cropping position.

[0559] In step S4005, the overall control CPU 101 stores the movement speed and subtraction amount of the cutting position obtained in step S4004 in the primary memory 103 and exits this subroutine.

[0560] In step S4006, the overall control CPU 101 calculates an expected value by allocating the sum of the subtraction amounts stored in the primary memory 103 so that the change in the movement speed of each cropping position over a certain past period, also stored in the primary memory 103, remains constant, and then exits this subroutine. The certain past period may be the period from when the cropping position actually started moving until the present, or the period from when the angular velocity sensor 107 detected the movement of the camera body 1 until the present. Alternatively, to simplify the process, it may be set to a fixed period of approximately 0.5 to 3 seconds. The expected value for periods prior to the above certain past period is set to the movement speed of the cropping position obtained in step S4004.

[0561] Table 1 below shows the displacement (velocity) of the data graphed in Figures 27(a) to (f). Specifically, the movement velocity of the cutting position determined in step S304 is shown in Table 1(c), and the gyro movement velocity calculated in step S4002 is shown in Table 1(d). In addition, the expected value calculated in step S4006 is shown in Table 1(e).

[0562] [Table 1]

[0563] The subroutine for cropping range correction shown in Figure 28(b) will be explained using the example of a user who initially remains still facing forward and gradually looks to the right, as shown in Table 1.

[0564] Initially, the user is looking forward, so the gyro movement speed calculated in step S4002 is approximately 0° / second. In other words, in step S4003, it is determined that the gyro movement speed has not decreased, and the process proceeds to step S4004. In this case, the position of the face also does not change, so the movement speed of the cropping position is also 0° / second. Furthermore, the subtraction amount calculated in step S4004 is also 0° / second.

[0565] After about 1 second, the user begins to turn to the right, but due to the time lag caused by the low-pass filter, the movement speed of the cropping position is still 0° / second, as shown in Figure 27(c). On the other hand, as shown in Figure 27(d), the camera body 1 has not yet moved, meaning the gyro movement speed is also approximately 0° / second. Therefore, just as when the user is still stationary in front, the subtraction amount calculated in step S4004 is also 0° / second.

[0566] As the user turns further to the right, and after about 2 seconds, the movement speed of the cropping position becomes 10° / second, as shown in Figure 27(c). On the other hand, as shown in Figure 27(d), the camera body 1 has not yet moved, meaning the gyro movement speed is still approximately 0° / second. Therefore, the subtraction amount calculated in step S4004 is 10° / second.

[0567] As the user turns further to the right, after about 4 seconds, the user's body also begins to turn to the right. That is, as shown in Figure 27(d), the orientation of the camera body 1 changes, so the gyro movement speed becomes 10° / second. As the body begins to rotate, as shown in Figure 27(b), the actual angular velocity of the face decreases by the relative velocity between the camera body 1 and the face. However, due to the time lag caused by the low-pass filter, the movement speed of the cropped position shown in Figure 27(c) is still 10° / second at this point. Therefore, taking this time lag into account, the subtraction amount calculated in step S4004 is 10° / second.

[0568] As the user turns further to the right, the gyro movement speed remains at 10° / second (Figure 27(d)) ​​from about 5 seconds onward, but the movement speed of the cutting position shown in Figure 27(c) decelerates to 0° / second. Therefore, the subtraction amount calculated in step S4004 is -10° / second.

[0569] Although not shown in Figure 27, when the user finishes turning to the right after about 6 seconds, the gyro movement speed becomes 0° / second, and only then does the process proceed to step S4006. In this case, the total subtraction amounts calculated so far and stored in the primary memory 103 becomes +10° / second. This total subtraction amount is allocated to the primary memory 103 so that the change in movement speed of each cutout position over a certain period in the past remains constant, and the expected value is calculated. Here, the movement speed of the cutout position shown in Figure 27(c) is 10° / second, 10° / second, 10° / second, and 0° / second from the start of acceleration to the present (2 to 6 seconds), as shown in Table 1. Therefore, in order to keep the change in movement speed of each cutout position constant (no change in this case), the expected value for the period from 2 to 6 seconds is set to 10° / second for all of them.

[0570] In this embodiment, explanations were given at one-second intervals for simplicity, but typically, the frame rate for video capture is 24-60 fps. On the other hand, face direction detection and gyroscope detection often do not need to be performed 60 times per second, so it is preferable to change the timing of face direction detection processing and cropping range correction processing from the timing of image capture. For example, image capture may be performed at 60 fps, but the timing of face direction detection processing and cropping range correction processing can be performed at 10 fps without any problems, and can be changed as appropriate considering the application and power consumption.

[0571] As explained above, this embodiment demonstrates an example where the speed of movement in the observation direction can be kept constant, so that when the observation direction changes significantly, the movement of the face and the movement of the body (camera body) combine, preventing a change in the field of view movement speed in the video and resulting in a poor-looking video.

[0572] In this embodiment, an example of cropping an ultra-wide-angle image according to the observation direction is shown, but the embodiment is not limited to this. For example, the overall control CPU 101 (imaging direction changing means) may change the imaging direction of the imaging unit 40 according to the observation direction. However, in this case, the camera body 1 must be provided with a mechanism (driving means) that mechanically drives the imaging direction of the imaging unit 40, specifically the orientation of the imaging lens 16 and the solid-state image sensor 42, in the yaw and pitch directions in a manner not shown.

[0573] Furthermore, although this embodiment demonstrates the smoothing of the face direction detection results, it is preferable to perform similar processing when the overall control CPU 101 (vibration isolation means) performs the vibration isolation control described in Embodiment 1, as this also causes a delay in tracking the face direction.

[0574] (Example 5) In Example 5, a method for reducing the difference between the user's field of view and the secondary recorded video (hereinafter referred to as "recorded video") caused by parallax resulting from the positional difference between the user's eye position and the mounting position of the shooting / detection unit 10 will be explained using Figures 29 to 34.

[0575] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 5, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0576] First, to aid understanding, we will explain the difference between the user's field of view and the recorded video that occurs in Example 1.

[0577] Figure 29 is a schematic diagram illustrating the relationship between the user's field of view and the target field of view in Example 1, when the observation target 5020 is a close-range subject.

[0578] Figure 29(a) is a schematic diagram showing the image 5900 including the object of observation 5020 as captured by the solid-state image sensor 42, and Figure 29(b) is a schematic diagram showing the positional relationship between the user 5010 and the object of observation 5020.

[0579] As shown in Figure 29(b), when the object of observation 5020 is below the height of the user's eyes 5011, the user's face direction 5015 is directed downwards. At this time, the object of observation 5020 with a background such as a floor (not shown) is visible in the user's field of view.

[0580] In Example 1, the observation direction 5040 (Figure 29(b)) parallel to the user's face direction 5015 detected by the face direction detection unit 20 is set as the recording direction. Therefore, as shown in Figure 29(b), if the observation target 5020 is a close-range subject, a problem arises in that the target field of view 5045 will be set to an area that does not include the observation target 5020.

[0581] In such cases, even if the background visible to the user 5010 (such as the floor, not shown) and the background captured by the imaging / detection unit 10 (such as the ceiling, not shown) are different, the recording direction should be set to the direction 5030, where the area 5035 including the object of observation 5020 is the target field of view, rather than the observation direction 5040.

[0582] The above problem is caused by parallax resulting from the positional difference between the user's (5010) eye (5011) position and the mounting position of the imaging / detection unit (10). Therefore, in this embodiment, a parallax correction mode processing is performed to appropriately adjust the recording direction, which is set based on the user's (5010) face orientation, according to the parallax.

[0583] Figure 31 is a block diagram showing the hardware configuration of the camera body 1 according to this embodiment.

[0584] The hardware configuration of the camera body 1 in this embodiment differs from the camera body 1 of Embodiment 1 shown in Figure 5 only in the presence of a distance measuring sensor 5100. While there are no particular limitations on the placement of the distance measuring sensor 5100 in the camera body 1, in this embodiment, as shown in Figure 30, the distance measuring sensor 5100 is provided on the outer edge of the stop switch 15.

[0585] The distance measuring sensor 5100 is a sensor that measures the distance to an object. The configuration of the distance measuring sensor 5100 is not particularly limited. In this example, the distance measuring sensor 5100 is an active type sensor that projects infrared light, laser, millimeter waves, etc., onto an object and measures the distance to the object by its reflection. Alternatively, the distance measuring sensor may be a passive type sensor that measures the distance to an object based on the phase difference of light rays transmitted through the imaging lens 16.

[0586] The distance measuring sensor 5100 is connected to the overall control CPU 101 and controlled by the overall control CPU 101.

[0587] Figure 32 is a schematic diagram illustrating the relationship between the user, the calibrator 850, and the target field of view 5080 during calibration, including parallax correction mode processing, in this embodiment.

[0588] Figure 32(a) is a schematic diagram showing the image 5900 including the calibrator 850 as projected onto the solid-state image sensor 42, and Figure 32(b) is a schematic diagram showing the positional relationship between the user 5010 and the calibrator 850.

[0589] The target field of view 5080 in Figure 32(a) is the target field of view when calibration, including the parallax correction process described later, has not been performed, and the face direction 5015 detected by the face direction detection unit 20 is facing forward.

[0590] On the other hand, the target field of view 5090 in Figure 32(a) is the target field of view when calibration including the parallax correction process described later has been performed and the face direction 5015 detected by the face direction detection unit 20 is facing forward.

[0591] Figure 33A is a flowchart of the parallax correction mode processing, which is part of the preparation process for step S100 in Figure 7A in this embodiment. The details of this process will be explained below using Figures 32(a) and (b).

[0592] In the preparation operation process of step S100 in Figure 7A related to this process, when the parallax correction mode is activated by the user 5010's operation on the calibrator 850 (step S5101), the display device control unit 801 displays the positioning index 851 (step S5102).

[0593] Next, the display control unit 801 instructs the user to position (specified position) where the calibrator 850 should be held. Specifically, the display control unit 801 instructs the user 5010 to hold the positioning indicator 851 in front of them at eye level by displaying an instruction 855 similar to that in Figure 22A (step S5103).

[0594] When the instruction display 855 is shown, the user 5010 holds the calibrator 850 over the designated position indicated in step S5103 and directs their face direction 5015 towards the positioning indicator 851 (front). At this time, the user 5010, the positioning indicator 851, and the imaging / detection unit 10 are in the positional relationship shown in Figure 32(b).

[0595] Subsequently, when the display device control unit 801 determines that the user has viewed the positioning index center 852 in the center of the field of view, it measures the distance 5050 (Figure 32(b)) between the imaging / detection unit 10 and the positioning index 851 using the distance measuring sensor 5100 (step S5104).

[0596] Next, the overall control CPU 101 detects the horizontal axis 5060 of the imaging and detection unit 10 using the angular velocity sensor 107 (attitude detection means) (step S5105). This identifies the horizontal position 5065 of the image 5900 (Figure 32(a)) captured by the solid-state image sensor 42.

[0597] Furthermore, in step S5105, the overall control CPU 101 obtains the distance 5855 (Figure 32(a)) between the center of the positioning index 851 on the image 5900 and the horizontal position 5065. Subsequently, the overall control CPU 101 (angle calculation means) calculates the angle 5055 (Figure 32(b)) between the horizontal axis 5060 and the direction of the positioning index 851 as seen from the imaging / detection unit 10. This angle 5055 is calculated using the distance 5855 and information about the relationship between a certain point on the image 5900 and the angle of incidence of light imaged at that point. This information is stored in memory (for example, built-in non-volatile memory 102).

[0598] Subsequently, the overall control CPU 101 (vertical distance calculation means) uses the distance 5050 and the angle 5055 calculated in step S5105 to calculate the vertical distance 5070 between the shooting / detection unit 10 and the user's eye 5011 (step S5106), and then exits this subroutine.

[0599] Here, a method for measuring the vertical distance 5070 between the imaging / detection unit 10 and the user's eye 5011 is described using a method different from that of Example 2, but the method is not limited to this. For example, the vertical distance 5070 between the imaging / detection unit 10 and the user's eye 5011 may be measured using the method described in Example 2, or the user 5010 may be directly asked to input the value of the vertical distance 5070.

[0600] The calibration process, including the parallax correction mode processing, in this embodiment is basically the same as the processes in steps S3101 to S3111 in Figure 21 that are performed in Embodiment 2, so its explanation is omitted.

[0601] However, in step S3110, in addition to the processing described in Example 2, parallax correction is performed based on the vertical distance 5070 (Figure 32(b)) calculated in the parallax correction mode processing in Figure 33A. That is, calibration is performed so that the user's field of view 5010 and the target field of view 125 coincide at infinity.

[0602] Figure 33B is a flowchart of the recording direction and range determination subroutine S300 described in Figure 7A in this embodiment. This process will be described below with reference to Figure 34 as well. Steps in Figure 33B that overlap with those in Figure 7D are denoted by the same reference numerals, and redundant explanations are omitted.

[0603] In Figure 33B, first, the overall control CPU 101 acquires distance information of the image capture range (imaging area) using the distance measuring sensor 5100 (distance measuring means) (step S5301).

[0604] Next, the overall control CPU 101 (creation means) creates a defocus map 5950 (Figure 34(a); distance map information) based on the distance information (measurement result by the distance measuring sensor 5100) obtained in step S5301 (step S5302).

[0605] The defocus map 5950 in Figure 34(a) represents the defocus map created when imaging was performed with the observation target 5020 floating in the room, as shown in Figure 34(c). Here, in order to clearly show the distance information in the defocus map 5950, it is represented in six distance areas (1) to (6) from the closest distance from the imaging / detection unit 10. However, in reality, the defocus map can also be created without any steps.

[0606] Next, the overall control CPU 101 calculates the direction of the observation target 5020 as seen from the imaging / detection unit 10 based on the defocus map 5950, face direction 5015, and vertical distance 5070 (Figure 32(b)) (step S5303). In other words, parallax correction is performed for the observation direction set based on the face direction.

[0607] After that, the process in steps S301 to S305 in Figure 7D is performed, and then the subroutine is exited.

[0608] By using the defocus map 5950 created in this way and the detection result of the face direction 5015, it is possible to calculate the direction of the observation target 5020 as seen from the imaging / detection unit 10. However, due to the parallax explained using Figure 29, it is not possible to measure only the distance to the observation target 5020 with the distance measuring sensor 5100 without creating the defocus map 5950.

[0609] The degree of parallax effect described in this embodiment varies depending on the distance between the user 5010 and the object being observed. That is, for objects being observed that are a certain distance away from the user 5010, the effect of parallax can be ignored, so even in the recording direction and range determination process in Embodiment 1, it is possible to crop and record the image with a target field of view that includes the object being observed. For example, if the user 5010 observes an object being observed 5021 (Figure 34(c)) located in a medium-distance area (5) that is a certain distance or more away from the user 5010, it is not necessary to perform parallax correction of the recording direction in step S5303. This is because the target field of view 5043 (Figure 34(b)), which is set according to the recording direction 5041 (observation direction) inferred based on the face direction 5016 detected by the face direction detection unit 20, includes the object being observed 5021.

[0610] On the other hand, according to this embodiment, the distance range between the user and the object being observed, in which the user 5010 can keep the object within the target field of view, can be expanded to a closer distance than in Embodiment 1. For example, suppose the user 5010 is observing an object being observed 5020 (Figure 34(a)) located in a close-range area (1) that is close to the user 5010. In this case, in Embodiment 1, the observation direction 5040 (recording direction) is inferred based on the face direction 5015 detected by the face direction detection unit 20. The target field of view 5042 (Figure 34(b)) set according to this observation direction 5040 does not include the object being observed 5020. However, in this embodiment, parallax correction is performed for the observation direction 5040 in step S5303 of Figure 33B, and the target field of view 5036 including the object being observed 5020 is set according to the recording direction after this parallax correction. Therefore, it becomes possible to capture images of observation targets, such as observation target 5020, at a very close distance from the user 5010 where the effect of parallax cannot be ignored.

[0611] Furthermore, according to this embodiment, it becomes possible to record an observation target located in the medium-range area (5) closer to the center of the target field of view. For example, when the user 5010 is observing an observation target 5021 (Figure 34(a)) located in the medium-range area (5), if parallax correction of the recording direction 5041 is not performed as in Embodiment 1, the target field of view 5043 will be set with the observation target 5021 at the top. On the other hand, in this embodiment, parallax correction for the recording direction 5041 is performed in step S5303 of Figure 33B, and according to the recording direction after this parallax correction, a recording area 5037 is generated in which the observation target 5021 is at the center.

[0612] As shown above, by implementing the parallax correction in this embodiment, it becomes possible to capture the object of observation more precisely in the center of the cropped image compared to Embodiment 1.

[0613] In this embodiment, parallax correction is also performed during calibration to ensure that the user's field of view and the recording area coincide at infinity. Then, parallax correction is performed such that the closer the distance between the user and the observed object during imaging, the greater the deviation in the recording direction before and after correction. However, the parallax correction of this embodiment may also be performed for finite positions, for example, subjects at a distance greater than the position of the calibrator 850 relative to the user in the calibration process of Embodiment 2, or subjects at a close distance.

[0614] (Example 6) In Example 6, the method for determining the cutting range when the calculation of the observation direction fails will be explained using Figures 35, 36A, and 36B.

[0615] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 6, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0616] In Example 1, as shown in Figure 7A, in step S200, the target field of view is set in step S300 by the recording direction and range determination process based on the observation direction calculated from the face direction detected by the face direction detection unit 20. However, the face direction detection unit 20 may be covered by obstacles such as collars or hair, the face direction detection unit 20 may malfunction, or the face direction detection unit 20 may move away from the user. In such cases, the user's face direction cannot be obtained, and the image of the target field of view that the user wanted to capture cannot be captured.

[0617] In Patent Document 1, if the second camera capturing the user fails to detect the user, the fact that the user could not be detected is not saved in the history of the user's observation information, and the user detection is attempted again. Furthermore, when capturing images by tracking the direction of the face, if the detection of the direction of the face fails, the imaging direction is determined according to the situation, thereby capturing images that do not deviate significantly from the user's intentions.

[0618] In contrast, in this embodiment, if the user's face direction can be detected, the face direction detection unit 20 detects the face direction, as in Embodiment 1, and captures an image of the target field of view in the recording direction based on the observation direction calculated based on this. On the other hand, if the user's face direction cannot be detected and the user's observation direction cannot be calculated, an image of the target field of view that takes the user's intention into account is captured. That is, in this embodiment, when the face direction detection process is completed in step S200, the observation direction determination process is executed before the recording direction / range determination process is executed in step S300. In this process, if the face direction detection unit 20 fails to detect the user's face direction, the user's intention is determined according to the situation and the observation direction is estimated. That is, an image of the target field of view in the recording direction is captured based on information other than the observation direction calculated from the face direction.

[0619] Figure 35 is a flowchart of the observation direction determination process according to this embodiment, which is executed by the overall control CPU 101. This process will be explained below using Figures 36A and 36B.

[0620] First, in step S6001, the face direction detection unit 20 determines whether it has been able to acquire the face direction. If the face direction has been acquired, the process proceeds to step S6004, where the overall control CPU 101 (mode transition means) switches the mode of this process to the face direction mode (first imaging mode) and determines the observation direction calculated from the face direction using the method shown in Example 1 as the recording direction. After that, the process exits this subroutine.

[0621] On the other hand, if the face direction cannot be obtained (NO in step S6001), the overall control CPU 101 (mode transition means) proceeds to step S6002 to transition to another mode and determines whether or not there is a subject that has been tracked in the past.

[0622] Here, we will explain the determination process in step S6002 using Figure 36A, which shows the relationship between the detection state of the user's observation direction for each frame and the captured video.

[0623] In Figure 36A, n is the frame number of the video, θ is the horizontal movement angle of the user's face, and user state indicates the positional relationship between the user and the object of observation in each frame. The overall video shows the ultra-wide-angle video captured by the camera unit 40 in each frame, and the captured video shows the image secondarily recorded in each frame, corresponding to the dashed area of ​​the overall video.

[0624] Figure 36A illustrates a case where, as shown in each screen of the user state, the user is observing the object indicated by the "□" at the bottom of the screen, and n=5, or the 5th frame, the user's observation direction could not be detected.

[0625] In this embodiment, the current frame is used as the reference point, and the four frames preceding it are defined as a predetermined period. If the same subject is found to be present in the captured video three or more times during this predetermined period, it is determined that the subject was previously tracked.

[0626] As shown in Figure 36A, for n=1 to 4, even though the movement angle θ changes by +10° each time, the captured video contains an object represented by a rectangle "□" which can be identified as the same subject. Therefore, for n=5, it is determined that there is a subject that was previously tracked.

[0627] Furthermore, the judgment criteria in step S6002 may be changed in accordance with the face direction detection period and the accuracy of the face direction detection unit 20.

[0628] Returning to Figure 35, if it is determined that there is a subject (the same subject) that was tracked within the past predetermined time (YES in step S6002), proceed to step S6005.

[0629] In step S6005, the processing mode is switched to a pre-subject tracking mode (second imaging mode) where the pre-subject direction is set as the recording direction. After determining the recording direction to track the subject, the process proceeds to step S6008. Thus, in this embodiment, even if the face direction can no longer be detected, if there is a subject that was tracked in the past, the system switches to the pre-subject tracking mode to determine the recording direction, thus reflecting the user's intentions immediately before the recording. Note that the subject recognition method and subject tracking detection method in the captured video by the overall control CPU 101 (subject recognition means) are publicly known, so a detailed explanation is omitted.

[0630] On the other hand, if it is determined that there is no subject that has been tracked in the past (NO in step S6002), the process proceeds to step S6003.

[0631] In step S6003, it is determined whether a subject previously registered in the built-in non-volatile memory (subject registration means) has been detected in the most recent captured video.

[0632] In this embodiment, pre-registration of a subject is performed by the user specifying an image containing the person to be photographed from images stored in the display device 800, the display device control unit 801 recognizing the characteristics of the selected person, and transmitting this information to the overall control CPU 101 in the camera body 1. Note that the subject detected in step S6003 is not limited to this; for example, it may be a subject included in the captured video acquired at the readout completion timing or other detection timings. Furthermore, whether the pre-registered subject matches the subject in the most recent captured video is determined by a pattern matching method. Since the pattern matching method is publicly known, a detailed explanation is omitted.

[0633] If it is determined that a subject that was registered in advance has been detected in the latest recorded video (YES in step S6003), proceed to step S6006.

[0634] In step S6006, the processing mode is switched to the registered subject mode (third imaging mode), which sets the direction of the subject detected in step S6003 as the recording direction. After determining the recording direction to track the registered subject, the process proceeds to step S6008.

[0635] On the other hand, if it is determined that a pre-registered subject was not detected in the latest captured video (NO in step S6003), it is determined that it was not possible to estimate the subject to be observed, and the process proceeds to step S6007.

[0636] In step S6007, the overall control CPU 101 (angle of view changing means) switches the mode of this process to the subject-lost mode (fourth imaging mode), in which the recording direction before the failure to detect the face direction remains unchanged, while the imaging angle is changed to a wider angle by comparing it with a specified angle of view. Then, the process proceeds to step S6008. Note that in the subject-lost mode, the recording direction may be continuously moved by the amount of change in the observation direction before the failure to detect the face direction.

[0637] Here, we will explain the case where the system proceeds to step S6007, which is in subject loss mode, using Figure 36B.

[0638] Figure 36B illustrates the case where n=5, or the 5th frame, the user's observation direction could not be detected.

[0639] In the example in Figure 36B, no major subject was found for n=1 to n=4, and the pre-registered subject was not found in the video captured at n=5. Therefore, the observation direction for n=5 is shifted to the rightward direction on the full-screen image, which is the direction of movement for n=1 to n=4, with inertia. In addition, the field of view extracted from the full-screen image is changed to a wide-angle.

[0640] In step S6008, if the recording direction was determined from a direction other than the face direction in any of steps S6005 to S6007, the overall control CPU 101 (notification means) notifies the user of an error (detection error) indicating that face direction detection failed. After that, the subroutine is exited. In this embodiment, a warning is issued to the user using the vibrating body 106 shown in Figure 5. However, the notification method in step S6008 is not limited to the method in this embodiment, and other notification methods may be used, such as a warning using the LED 17 or a display device 800 or other terminal linked to the camera body 1.

[0641] As described above, in this embodiment, if the direction of the face cannot be detected, the recording direction and field of view are changed according to the situation, thus preventing the user from missing the image of the intended field of view that they originally wanted to capture.

[0642] In other words, in this embodiment, if the direction of a face cannot be detected, and a subject that has been tracked in the past or a pre-registered subject can be detected, the camera will track that subject. On the other hand, if such a subject cannot be detected, the field of view is changed to a wider angle than the standard field of view in order to prevent missed shots and to make it easier to re-detect the subject.

[0643] This prevents unintended images from being captured due to a failure in detecting the user's facial orientation.

[0644] Although the processing in steps S6001 to S6008 is performed every frame, the mode can be changed again after transitioning to each mode based on mode determination information, such as whether the face direction has been acquired from the face direction detection unit 20. For example, in this embodiment, if a pre-registered subject is detected as a result of widening the field of view in the subject-lost mode, the system switches to the registered subject tracking mode, which uses the direction of the detected subject as the observation direction. In this case, the widened field of view is returned to the standard field of view.

[0645] Furthermore, although the mode was changed in a single determination in this embodiment, the mode may be switched based on multiple results depending on the frame rate and face direction detection capability.

[0646] (Example 7) In Example 7, a method for determining the observation direction according to the accuracy (reliability) of face direction detection will be explained using Figures 37 to 40.

[0647] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 7, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0648] In Example 6, the mode for determining the observation direction was switched depending on whether or not the face direction could be detected, preventing imaging in a recording direction unintended by the user. On the other hand, as in Patent Document 1, if the user's face direction cannot be consistently and stably detected, imaging may occur at an angle of view unintended by the user. An example of a situation where the face direction cannot be consistently and stably detected is when the imaging / detection unit 10 of the camera body 1 is attached to the front of the clavicle, as shown in Figure 1B, in which case the accuracy of face direction detection may decrease due to the influence of the collar, hair, etc.

[0649] As shown in Figure 37, when the user is facing to the right (Figures 37(b), (c)) compared to when the user is facing forward (Figure 37(a)), a larger area of ​​the chin and cheeks is hidden by the body and shoulders. In other words, depending on the direction of the face, the area of ​​the face that can be used for detecting the direction of the face becomes smaller, and the detection accuracy is likely to decrease. This characteristic of the camera body 1 depends largely on the position in which the user is wearing the camera body 1.

[0650] Therefore, in this embodiment, the accuracy (reliability) of face direction detection is calculated according to the mounting position of the camera body 1 and the detection result of face direction. If the reliability is high, the face direction is reflected more in the observation direction, and if the reliability is low, other information is reflected more in the observation direction. This makes it possible to capture images while understanding the user's intentions.

[0651] Figure 38 is a flowchart of the observation direction determination process when acquiring face direction, according to this embodiment, which is performed instead of the process in step S6004 of Figure 35. This process is executed by the overall control CPU 101 (observation direction determination means).

[0652] First, in step S7001, the overall control CPU 101 (first observation direction calculation means, reliability calculation means) calculates the face direction θ acquired by the face direction detection unit 20 during the imaging of the nth frame. n Face direction reliability T based on (first observation direction) n Calculate.

[0653] Face direction reliability T n It is calculated as follows:

[0654] First, the direction of the face θ n The direction of the face θ yaw ,θ pitch ,θ roll It is divided into three components. Here, the face direction θ yaw θ represents the rotational component of the face moving from side to side, and the face direction θ. pitch This represents the rotational component that moves the face up and down, and the face direction θ. roll This represents a rotational component that moves as if tilting the head.

[0655] In this embodiment, the camera body 1 is attached to the user's clavicle and the direction of the face is detected from below the face, so Tn (0 ≤ Tn ≤ 1) can be calculated using the following equation 701.

[0656]

number

[0657] Figure 39 shows the face direction θ. yaw and facial direction reliability T n This shows the change in the value of the face direction θ. yaw The greater the change from the front view, the higher the facial direction reliability T n This indicates a decrease.

[0658] In this embodiment, the facial direction reliability T is calculated using Equation 701. nAlthough this was calculated, it is also possible to use a weighted average of face direction reliability calculated in the past, weighted according to the face direction detection accuracy of the face direction detection unit 20 and the detection frame rate. n When calculating the value, factors such as the accuracy of pattern matching and the mounting position may be weighted accordingly.

[0659] In this embodiment, the facial direction reliability for predicting the observation direction was calculated using Equation 701. However, the method for calculating facial direction reliability is not limited to this. For example, the facial direction reliability adjusted according to the mounting position that can be estimated by the calibration in Embodiment 2 may be used. Also, if the detection accuracy is judged to be low during calibration, the facial direction reliability may be changed according to that accuracy. Furthermore, when detecting facial direction using machine learning, the precision rate may be reflected in the facial direction reliability.

[0660] In step S7002, the overall control CPU 101 determines the angular velocity ω of the face movement. n This is determined by the face direction θ acquired by the face direction detection unit 20 during the acquisition of the nth frame. Specifically, the face direction θ acquired by the face direction detection unit 20 during the acquisition of the nth frame. n and face direction acquisition time t n And the face direction θ from the previous frame, stored in the primary memory 103. n-1 and the acquisition time t n-1 From angular velocity ω n This can be calculated using equation 702 below.

[0661]

number

[0662] In this embodiment, angular velocity ω n The angular velocity was calculated using the current frame and the information from the previous frame, but depending on the frame rate, one or more pieces of past information may be used to determine the angular velocity.

[0663] In step S7003, the overall control CPU 101 (observation direction prediction means) predicts the current face direction from past face direction transitions stored in the primary memory 103. In this embodiment, the current frame is used as the reference point, and the four frames prior to it are defined as a predetermined period. If the face direction has continuously shifted in a certain direction that can be determined to be the same three or more times during these four frames, it is determined that the observation direction can be predicted from the past face direction and angular velocity. Furthermore, when making this prediction, the predicted angular velocity ω is the weighted average of the angular velocities obtained from the past four frames. ave The predicted face direction θ is calculated using the following equation 703. ave The (second observation direction) is calculated using equation 704 below. The calculations in equations 703 and 704 correspond to the processes shown in Figures 40(a1) and (a2), respectively.

[0664] Furthermore, the length of the predetermined period used in step S7003 and the method of applying the weighted average may be changed according to the frame rate and the detection accuracy of the face direction detection unit 20.

[0665]

number

[0666] In step S7004, the overall control CPU 101 predicts the observation direction using internal information other than the information from the face direction detection unit 20, which is stored in the primary memory 103. Specifically, in this embodiment, it determines from the subject detection history whether it is currently tracking a subject. If it determines that it is tracking, it predicts the observation direction θ based on the subject's movement. sub The (second observation direction) is calculated. In this embodiment, the current frame is used as the reference point, and the four frames preceding it are defined as a predetermined period. If the same subject (same subject) is detected three or more times from the captured images of these four frames, the system is considered to be tracking the subject. The subject tracking determination criteria may be changed according to the detection cycle and detection accuracy of the overall control CPU 101. Since the subject tracking detection method is publicly known, a detailed explanation is omitted.

[0667] Still, in this embodiment, the internal information used for predicting the observation direction in step S7004 is not limited to the subject detection history. For example, according to the mounting position and performance of the camera body 1, the face information of the user reflected in the imaging unit 40, or the information on the movement and posture of the camera body 1 detected by the angular velocity sensor 107 and the acceleration sensor 108 may be used to predict the observation direction. Also, as in step S6006 of Embodiment 6, when there is a pre-registered subject, the overall control CPU 101 (the third observation direction prediction means) may obtain the direction of the pre-registered subject on the latest captured video as the predicted observation direction θ sub as well.

[0668] In step S7005, the overall control CPU 101 stores the face direction detection related information in the primary memory 103 as a history. Here, the face direction detection related information includes the angular velocity ω n of the face movement generated in step S7002, the face direction reliability T n calculated in step S7001, the face direction θ n detected by the face direction detection unit 20, the face direction acquisition time t n , and the information indicating the generation time of each of these pieces of information.

[0669] In step S7006, the overall control CPU 101 determines whether the face direction reliability T n calculated in step S7001 is greater than or equal to a predetermined value. If the face direction reliability T n is greater than or equal to the predetermined value, it is determined that the reliability of the face direction is high, and the process proceeds to step S7009.

[0670] In step S7009, the overall control CPU 101 determines the face direction as the current observation direction θ' n and proceeds to step S7013.

[0671] On the other hand, if the face direction reliability T n calculated in step S7001 is less than the predetermined value (NO in step S7006), the process proceeds to step S7007.

[0672] In step S7007, in step S7003, the predicted face direction θave It is possible to estimate |θ, and n -θ ave If the condition that | is within a predetermined angle is met, proceed to step S7010. In this embodiment, the determination is made using the predetermined angle π / 8.

[0673] In step S7010, the overall control CPU 101 (first observation direction prediction means) determines θ n and θ ave and facial direction reliability T n Using the current observation direction θ' n Determine the current observation direction θ'. In this example, the current observation direction θ' is determined. n ′ is calculated using the following equation 705, and the process proceeds to step S7013. The calculation in equation 705 corresponds to the process shown in Figure 40(b). As shown in Figure 39, the face direction θ yaw The smaller the absolute value of the angle, the lower the face direction reliability T. n As the value increases, therefore, the face direction θ yaw If the absolute value of the angle is small, the current observation direction θ' n As shown in equation 705, the face direction θ n This is reflected more. On the other hand, the face direction θ yaw If the absolute value of the angle is large, the current observation direction θ' n As shown in equation 705, the face direction θ yaw Other information (predicted face direction θ) ave ) is reflected more.

[0674]

number

[0675] If the above conditions are not met in step S7007, proceed to step S7008 and predict the observation direction θ. sub It is possible to estimate |θ, n -θ sub If the condition that | is within a predetermined angle is met, proceed to step S7011. Similar to step S7010, in this embodiment the determination is made using π / 8 as the predetermined angle.

[0676] In step S7011, the overall control CPU 101 (second observation direction prediction means) determines the face direction θ n And, the predicted observation direction θ sub And, facial direction reliability T n Using the current observation direction θ' n Determine the current observation direction θ'. In this example, the current observation direction θ' is determined. n The value is calculated using equation 706 below, and the process proceeds to step S7013. Similar to step S7010, the face direction θ is as shown in Figure 39. yaw The smaller the absolute value of the angle, the lower the face direction reliability T. n Therefore, when the absolute value of the angle of the face direction θyaw is small, the current observation direction θ' n As shown in equation 706, the face direction θ n This is reflected more. On the other hand, the face direction θ yaw If the absolute value of the angle is large, the current observation direction θ' n As shown in equation 706, the face direction θ yaw Other information (predicted observation direction θ) sub ) is reflected more.

[0677]

number

[0678] If the above conditions are not met in step S7008, it is determined that a reliable observation direction cannot be obtained at present, and the process proceeds to step S7012.

[0679] In step S7012, the previous observation direction θ' n-1 The current observation direction θ' is moved in the observation direction with inertia based on the displacement in the past observation direction. n The camera determines the angle of view and sets it wider than the specified value, then proceeds to step S7013. This reduces the possibility of the user missing the intended subject.

[0680] In this example, facial direction reliability T n And the current observation direction θ' depending on the detection status of the subject. n The calculation method has been changed, but is not limited to this. For example, the predicted face direction θave and predicted observation direction θ sub When calculating each of these, the confidence level (prediction direction confidence level) is also calculated, and the calculated observation direction θ' is determined according to the calculated confidence levels. n You may also try to correct it.

[0681] Furthermore, if each of the calculated confidence values ​​falls below a predetermined value, there is a higher possibility that the user will miss capturing the intended subject. Therefore, it is preferable to set the field of view to a wider angle than the specified field of view. In this case, the process may proceed to step S7012. Subsequently, if any of the calculated confidence values ​​exceed a predetermined value, it is preferable to return the field of view to the specified field of view.

[0682] The processing shown in Figure 38 results in a facial direction reliability T n When the face direction θ is high, n current observation direction θ' n This is the decision. On the other hand, facial direction reliability T n If the facial direction reliability T is low, depending on the situation, n Information from when the value was high, and information other than face direction, are used to determine the current observation direction θ'. n Determine the recording direction, and if necessary, widen the field of view.

[0683] In other words, in this embodiment, the facial direction reliability T n If the value is low and the accuracy of face direction detection is expected to be low, predict the face direction θ ave or predicted observation direction θ sub By using this method, it is possible to prevent the capture of unintended images due to failures in face direction detection.

[0684] (Example 8) In Example 8, a method for mounting the camera body 1 in a stable position will be explained using Figures 41 to 45.

[0685] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 8, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0686] First, we will explain how to adjust the angle of the connection parts 80L and 80R (neck-hanging means).

[0687] Figure 41 is an enlarged view showing the imaging / detection unit 10 from the side. The following explanation uses the left connection section 80L as an example, but the right connection section 80R is adjusted in the same manner.

[0688] Figure 41(a) shows the connection part 80L in the standard position Ax0, and Figure 41(b) shows the connection part 80L in position Ax1, which is rotated by an angle θA1 with respect to the standard position Ax0 around the rotation axis OA. Figure 41(c) is a schematic diagram showing the internal mechanical mechanism of the angle holding part 81L that can be seen when the outer casing of the angle holding part 81L is removed.

[0689] As shown in Figure 41(c), an angle adjustment mechanism 8100 (neck-hanging angle adjustment means) is arranged inside the angle holding section 81L.

[0690] The angle adjustment mechanism 8100 consists of an angle adjustment cam 8101 that adjusts the angle of the angle holding unit 81L relative to the imaging / detection unit 10, and a locking member 8102 that locks the angle adjustment cam 8101. The rotation axis OA of the angle holding unit 81L coincides with the center of the angle adjustment cam 8101.

[0691] The locking member 8102 is biased against the angle adjustment cam 8101 by a spring (not shown), but while the angle adjustment button 85L (Figure 2F) is pressed, the biasing is released and it separates from the angle adjustment cam 8101. In other words, only while the angle adjustment button 85L is pressed, the angle holding part 81L of the connecting part 80L becomes rotatable relative to the shooting / detection part 10.

[0692] The user can adjust the connection part 80L from the standard position Ax0 (Figure 41(a)) to position Ax1 (Figure 41(b)) by pressing the angle adjustment button 85L and rotating the angle holding part 81L relative to the shooting / detection part 10.

[0693] In this embodiment, a stepped adjustment mechanism using an angle adjustment cam 8101 and a locking member 8102 is used as the mechanism for maintaining the angle of the angle holding unit 81L relative to the imaging / detection unit 10. However, a mechanism that enables stepless adjustment using sliding resistance may also be used.

[0694] Furthermore, in this embodiment, the user rotates the angle holding part 81L while pressing the angle adjustment button 85L, but the invention is not limited to this configuration. For example, a balanced configuration is possible in which the angle holding part 81L rotates when an external force exceeding a certain threshold is applied to the angle holding part 81L, without requiring the angle adjustment button 85L. For example, a ball could be used instead of the locking member 8102, or a sliding resistance could be used.

[0695] Figure 42 is a side view showing the camera body 1 attached to the user.

[0696] Figure 42(a) shows a user wearing the camera body 1 with the connector 80L in the standard position Ax0 and the band 82L adjusted to a longer length. Figure 42(b) shows a user wearing the camera body 1 with the connector 80L in the standard position Ax0 and the band 82L adjusted to a shorter length. Figure 42(c) shows a user wearing the camera body 1 with the connector 80L in position Ax1 and the band 82L adjusted to a shorter length.

[0697] As shown in Figures 42(a) and (c), if the relationship between the position of the connector 80L and the length of the band 82L is suitable for the user, the imaging lens 16 will face the user directly. On the other hand, as shown in Figure 42(b), if the relationship between the position of the connector 80L and the length of the band 82L is not suitable for the user, the imaging lens 16 will not face the user directly. In this case, Figure 42(b), the optical axis of the imaging lens 16 is pointing upward.

[0698] Since the connection section 80L is configured to allow for position adjustment, the user can attach the camera body 1 so that the optical axis of the imaging lens 16 is approximately parallel to the user's natural line of sight. Of course, if the appropriate mounting position for the camera body 1 for the user is when the optical axis of the imaging lens 16 is horizontal, then appropriate mounting is also possible.

[0699] Next, we will explain how to adjust the angle of the chest connection pads 18a and 18b.

[0700] Figure 43 is an enlarged view of the imaging / detection unit 10 from the side when the connection parts 80L and 80R are hidden. In the following explanation, the left chest connection pad 18a will be used as an example, but the right chest connection pad 18b will be adjusted in the same way.

[0701] Figure 43(a) shows the chest connection pad 18a in the standard position Bx0, and Figure 43(b) shows the chest connection pad 18a in position Bx1, which is rotated by an angle θB1 relative to the standard position Bx0 around the rotation axis OB. Figure 43(c) is a schematic diagram showing the internal mechanical mechanism of the imaging / detection unit 10 that can be seen when the exterior of the imaging / detection unit 10 is removed.

[0702] As shown in Figure 43(c), an angle adjustment mechanism 8200 (grounding angle adjustment means) is located inside the imaging and detection unit 10.

[0703] The angle adjustment mechanism 8200 consists of an angle adjustment cam 8201 for adjusting the angle of the chest connection pad 18 relative to the imaging / detection unit 10, and a locking member 8202 for locking the angle adjustment cam 8201. Point OB shown in Figures 43(a) to (c) is the pivot point of the chest connection pad 18a.

[0704] The locking member 8202 is biased against the angle adjustment cam 8201 by a spring (not shown), but while the angle adjustment button 8203 is pressed, the biasing is released and it moves away from the angle adjustment cam 8201. In other words, the chest connection pad 18 becomes rotatable relative to the imaging / detection unit 10 only while the angle adjustment button 8203 is pressed.

[0705] The user can adjust the position of the chest connection pad 18 from position Bx0 to position Bx1 by rotating the chest connection pad 18 relative to the imaging / detection unit 10 while pressing the angle adjustment button 8203.

[0706] In this embodiment, a stepped adjustment mechanism using an angle adjustment cam 8201 and a locking member 8202 is used as a mechanism to maintain the angle of the chest connection pad 18 relative to the imaging / detection unit 10. However, a mechanism that enables stepless adjustment using sliding resistance may also be used.

[0707] Furthermore, in this embodiment, the user rotates the chest connection pad 18 while pressing the angle adjustment button 8203, but the invention is not limited to this configuration. For example, a configuration that does not require the angle adjustment button 8203 and balances the chest connection pad 18 so that it rotates when an external force exceeding a certain threshold is applied to it, such as using a ball instead of the locking member 8202 or using sliding resistance, is also possible.

[0708] Figure 44 is a side view showing the camera body 1 attached by the user, with the connection part 80L hidden.

[0709] Figure 44(a) shows a user with their chest upright wearing the camera body 1 with the chest connection pad 18a in the standard position Bx0. Figure 44(b) shows a user with their chest lying down wearing the camera body 1 with the chest connection pad 18a in the standard position Bx0. Figure 44(c) shows a user with their chest lying down wearing the camera body 1 with the chest connection pad 18a in position Bx1.

[0710] As shown in Figures 44(a) and (c), if the position of the chest connection pad 18a is suitable for the tilt of the user's chest, the chest connection pad 18a will make contact with the user's chest over a wide area. On the other hand, as shown in Figure 44(b), if the position of the chest connection pad 18a is not suitable for the tilt of the user's chest, the chest connection pad 18a will only make contact with the user's chest over a small area. As shown in Figure 44(b), if the area in contact with the user's chest of the chest connection pad 18a becomes small, the shooting / detection unit 10 will easily move relative to the user's body when the user moves, causing the captured image to become significantly blurred.

[0711] Since the chest connection pad 18a is configured to allow for angle adjustment, the user can attach the camera body 1 so that the chest connection pad 18a is positioned over a wide area of ​​their chest, thereby suppressing blur in the captured image.

[0712] In this embodiment, the chest connection pad 18a is located in the imaging / detection unit 10, but the same effect can be obtained even if it is located in the connection unit 80L. In this case, for example, a mechanism similar to the angle adjustment mechanism 8100 shown in Figure 41(c) is located inside the connection unit 80L to adjust the angle of the chest connection pad 18a relative to the connection unit 80L.

[0713] Next, the configuration of the band section 82L and the electrical cable 84 will be described in detail.

[0714] As described in Example 1, the battery unit 90 (power supply) and the shooting / detection unit 10 of the camera body 1 are separate modules that are electrically connected via an electrical cable 84.

[0715] If the electrical cable 84 and the band portion 82L are separate components, it is undesirable from an aesthetic standpoint for the camera body 1, and it is also undesirable in that it becomes cumbersome for the user when putting it around their neck. Therefore, it is desirable that the band portion 82L and the electrical cable 84 be integrated, but the configuration is not limited to that shown in Figure 2B.

[0716] Figure 45 shows the band portion 82L and the connection surface 83L, which is a cross-section of the electrical cable 84 that is integrally formed therewith.

[0717] Figures 45(a) to (c) show the case where the electrical cable 84 is made of a flexible printed circuit board (FPC), and Figures 45(d) to (g) show the case where the electrical cable 84 is made of a thin wire cable.

[0718] In Figures 45(a) and (d), the electrical cable 84 is embedded inside the band portion 82L as viewed from the connection surface 83L. In this case, the material of the band portion 82L is preferably an injection-molded elastic material such as silicone rubber, elastomer, rubber, or plastic. One method for manufacturing a component in which the band portion 82L and the electrical cable 84 are integrated is to insert the electrical cable 84 during the injection molding of the band portion 82L. Alternatively, a manufacturing method may be adopted in which the band portion 82L is composed of two parts, the electrical cable 84 is sandwiched between them, and the two parts of the band portion 82L are integrated into a single component by adhesive or heat welding. However, as shown in Figures 45(a) and (d), it is sufficient that a component in which the band portion 82L and the electrical cable 84 are integrated can be manufactured, and the method is not limited to the two methods described above.

[0719] Figures 45(b), (c), and (e) show that the electrical cable 84 is in contact with the outside of the band portion 82L when viewed from the connection surface 83L.

[0720] In Figure 45(b), the band portion 82L, as viewed from the connection surface 83L, does not have any special shape for integrating with the electrical cable 84. Since the electrical cable 84 is simply bonded to the surface of the band portion 82L, it can be manufactured at low cost. If the electrical cable 84 (in this case, an FPC) is on the visible side, the appearance can be improved by painting the FPC or covering it with a film. Also, in the configuration of Figure 45(b), if the electrical cable is on the side that comes into contact with the user's neck, the wearing comfort can be improved by painting the electrical cable or covering it with a film.

[0721] In Figures 45(c) and (e), the band portion 82L, viewed from the connection surface 83L, is provided with a recessed shape 83a to integrate with the electrical cable 84, and the electrical cable 84 is positioned inside this recessed shape 83a. In this case, if the recessed shape 83a is provided on the side that contacts the user's neck, the aesthetic appearance of the camera body 1 is ensured, and if the recessed shape 83a is provided so that the electrical cable 84 does not directly contact the user's neck, a good fit for the user can be maintained without any special processing. Furthermore, if the band portion 82 is properly designed during manufacturing, there is no additional cost for adding the recessed shape 83a, making it cost-effective.

[0722] In Figures 45(f) and (g), as with Figures 45(a) and (d) mentioned above, the electrical cable 84 is embedded inside the band portion 82L when viewed from the connection surface 83L. Figure 45(f) shows the configuration when there is one electrical cable 84, and Figure 45(g) shows the configuration when there are multiple electrical cables 84. These configurations differ from Figures 45(a) and (d) in that they ensure a sufficient cross-sectional area of ​​the band portion 82L at the connection surface 83L. The cross-sectional area of ​​the band portion 82L at the connection surface 83L affects the torsional rigidity and bending rigidity of the band portion 82L, and these rigidities affect the stability of the shooting / detection unit 10 so that it remains stable in a fixed position on the body when the user wears the camera body 1. In other words, the larger the cross-sectional area of ​​the band portion 82L at the connection surface 83L, and the stronger the torsional rigidity and bending rigidity, the better the stability of the shooting / detection unit 10. It is preferable that the protruding side of the electrical cable 84 be located on the external side to ensure a good fit. The shapes shown in Figures 45(f) and (g) show the protruding shape of the electrical cable 84 in the band portion 82L, which is visible from the outside, but this shape is suitable from the viewpoint of ensuring the rigidity of the band portion 82L.

[0723] Based on the above, the shapes shown in Figures 45(c) and (e) are preferable in terms of balancing aesthetics and wearing comfort. However, if cost or rigidity is a priority, other shapes shown in Figure 45 can be used.

[0724] (Example 9) In Example 9, a modified example of the camera system including the camera body 1 will be described using Figures 46A and 46B.

[0725] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 9, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0726] In Example 1, a standard smartphone was used as the display device 800. However, there are many smartphones on the market, and their processing power varies widely. For example, the display device 800 in Example 1 has relatively high processing power. Therefore, when the camera body 1 transfers the video image in the recording direction, extracted from the ultra-wide-angle image, to the display device 800, information necessary for optical correction processing and image stabilization processing is added to the video image. Based on this information, the display device 800 performs distortion correction and image stabilization processing. On the other hand, a smartphone used by the user as the display device 800 may have insufficient processing power to perform these corrections. This embodiment assumes such a case.

[0727] The camera system of this embodiment includes a camera body 1' including an imaging device and a display device 9800 with lower processing power than the display device 800.

[0728] In camera body 1', once the initial video recording process (steps S100 to S600 in Figure 7A) is completed, the transfer process to the display device 9800 is not performed, and the processes in steps S800 and S900 are executed. Subsequently, camera body 1' transfers the video, for which the processes in steps S800 and S900 have been completed, to the display device 9800.

[0729] On the other hand, the display device 9800 records the video from the camera body 1' directly without performing the processing in steps S800 and S900.

[0730] The camera system of this embodiment will be described in detail below.

[0731] Figure 46A is a block diagram showing the hardware configuration of the display device 9800 connected to the camera body 1', which serves as the imaging device in this embodiment.

[0732] In Figure 46A, components identical to those in the hardware configuration of the display device 800 according to Embodiment 1 shown in Figure 6 are denoted by the same reference numerals, and redundant explanations are omitted.

[0733] The display device 9800 differs from the display device 800 in that it has a display device control unit 9801 instead of the display device control unit 801, and does not have a face sensor 806.

[0734] The display control unit 9801 is configured with a CPU that has lower processing power than the CPU that constitutes the display control unit 801 (Figure 6). Furthermore, the capabilities of the built-in non-volatile memory 812 and primary memory 813 may be lower than those in Example 1.

[0735] Figure 46B is a functional block diagram of the camera body 1'.

[0736] In Figure 46B, components identical to those in the camera body 1 according to Embodiment 1 shown in Figure 4 are denoted by the same reference numerals, and redundant explanations are omitted.

[0737] The functional blocks shown in Figure 46B differ from those in Figure 4 in that they include an optical correction / vibration damping processing unit 9080 that performs optical correction processing and vibration damping processing, and that each functional block is executed under the control of the overall control CPU 9101 rather than the overall control CPU 101. Another difference is that one of the predetermined communication partners of the transmission unit 70 is the display device 9800 rather than the display device 800.

[0738] In other words, in this embodiment, the optical correction / vibration damping processing unit 9080 of the overall control unit CPU 9101 performs optical distortion correction and vibration damping processing using optical correction values ​​and gyro data. Therefore, compared to the video file 1000 that the transmission unit 70 transfers to the display device 800 in Embodiment 1, the video file after optical distortion correction and vibration damping processing that the transmission unit 70 transfers to the display device 9800 in this embodiment has a smaller data size.

[0739] Furthermore, since the display device 9800 does not perform the processing in steps S800 and S900, its processing power can be kept lower compared to the display device 800. In addition, the image captured by the camera body 1' can be viewed on a simple display device 900 (viewing unit) consisting of a smartwatch or the like.

[0740] (Example 10) In Example 10, a modified example of the camera system including the camera body 1 will be described using Figures 47 and 48. Basically, this example will be described as a derivative of Example 1. For this reason, among the components of the camera system in Example 10, components that are the same as those in the camera system of Example 1 will use the same reference numerals, and redundant explanations will be omitted. Different components will be described in detail as needed.

[0741] In Example 9, the camera body 1' was required to have high performance as the display device 9800 had low processing power. However, increasing the capabilities of the camera body can increase the cost of the overall control unit CPU and its peripherals, and may also lead to heat generation due to process load. Therefore, Example 10 describes a configuration in which the processing power of the camera body is reduced and the processing power of the display device is increased.

[0742] Figure 47 is a functional block diagram of a camera system including a camera body 1001 and a display device 1080, showing the configuration of this embodiment. Components identical to those in the functional block diagrams of camera body 1 and camera body 1' shown in Embodiment 1 in Figure 4 and Embodiment 9 in Figure 46B are denoted by the same reference numerals, and redundant explanations are omitted.

[0743] The functional block diagram shown in Figure 47 differs significantly from those in Figures 4 and 46B in that the recording direction / angle determination unit 1083, the image cropping / development processing unit 1084 which performs image cropping and development, and the optical correction / vibration damping processing unit 1085 which performs optical correction processing and vibration damping processing are all located in the display device 1080.

[0744] The camera body 1001 now includes a primary face image processing unit 1030 for processing face images detected by the face direction detection unit 20, a primary main video processing unit 1050 for processing main video captured by the shooting unit 40, and an image synthesis unit 1055 for combining these images. The display device 1080 now includes a recording direction / angle determination unit 1083 and an image cropping / development processing unit 1084, and an image separation unit 1082 has been added. In addition, the receiving unit 1081, which was omitted from the explanation in Examples 1 and 9 where the functions of the display device 1080 were simpler, has been added to the diagram.

[0745] The order of processing will be explained using the flowchart in Figure 48. For flows that perform the same or similar processing as the flow in Figure 7A, the step number will be assigned a number obtained by adding 10000 (i.e., a number with "10" appended to the first two digits), and the explanation will be omitted. In addition, as an aid to the explanation, Figure 48 indicates on the right side of each step which device shown in Figure 47 is performing that step. Specifically, steps S10100 to S10700 in Figure 48 are performed by the camera body 1001, and steps S10710 to S10950 are performed by the display device 1080.

[0746] In Figure 7A of Example 1, face direction detection was performed in step S200 after the preparation operation in step S100. However, in this example, after the preparation operation in step S10100, face capture in S10200 and main capture in S10400 are performed simultaneously. Next, in S10450, the two video data captured in S10200 and S10400 are combined. Various methods of combining are possible; the two videos may be combined into a single video file, or the two video data may be linked in a way that prevents frame misalignment.

[0747] In this embodiment, the explanation will continue based on an example of combining two videos into a single video file. In S10450, the primary recorded combined image is wirelessly transmitted to the display device 10180 in S10700.

[0748] Steps from step S10710 onward are executed on the display device 1080. In step S10710, the combined video from S10450 is separated again into the face capture video and the main capture video. Then, in step S10720, a face direction detection process is performed to infer the observation direction from the separated face capture video. Details of the face direction detection process are as described using Figure 7C, similar to Example 1.

[0749] In step S10730, the recording direction and range determination process is performed. In step S10750, using the recording direction and field of view information determined in step S10730, the main captured video separated in step S10710 is cropped, and the recording range development process is performed on that area. In step S10800, optical correction processing is performed on the video developed in step S10750. In step S10900, image stabilization processing is performed.

[0750] Of course, in this embodiment as well, the order of steps S10800 and S10900 can be reversed. In other words, the image stabilization process can be performed first, followed by optical correction.

[0751] In step S10950, the display device control unit (video recording means) performs secondary recording, recording the video, which has undergone optical correction processing and vibration damping processing in steps S10800 and S10900, into the large-capacity non-volatile memory 814, and then terminates this process.

[0752] In this embodiment, by sending a composite image of the main capture video and the face capture video in step S10700, processing on the camera body 1001 can be simplified, resulting in cost reduction and reduced heat generation. Alternatively, as in Embodiment 1, gyro data and attitude data output from the angular velocity sensor 107 and acceleration sensor 108 may also be sent to the display device 1080 during the transfer in step S10700.

[0753] (Example 11) In Example 11, a modified example of the camera system including the camera body 1 will be described using Figures 49 and 50. Basically, this example will be described as a derivative of Example 1, but since the basic configuration is similar to that of Example 10, the same reference numerals will be used for components of the camera system in Example 11 that are the same as those in Example 11, and redundant explanations will be omitted. Different components will be described in detail as needed.

[0754] Example 10 describes a configuration in which the processing power of the camera body is reduced and the processing power of the display device 1080 is increased. While this configuration can reduce the load on the overall control CPU, the amount of data transmitted from the transmission unit 70 increases, which can result in increased power consumption and other issues such as heat generation.

[0755] Furthermore, recent developments are underway to include circuits specifically designed for image processing in the overall control CPUs installed in camera bodies. For example, some CPUs are already capable of performing tasks such as face direction detection, which is required in this case, making it possible to implement the system while keeping costs down and power consumption low. Utilizing this, in this embodiment, the camera body 1101 performs face detection, determines the recording direction and field of view, transfers the resulting image with this data to the display device 1180, and performs image cropping and development processing within the display device 1180.

[0756] Figure 49 is a functional block diagram of a camera system including a camera body 1101 and a display device 1180, showing the configuration of this embodiment. Components identical to those in the functional block diagrams of camera body 1 and camera body 1' camera body 1001 shown in Embodiments 1, 9, and 10 in Figure 4 are denoted by the same reference numerals, and redundant explanations are omitted.

[0757] Unlike Figures 4 and 46B, this embodiment differs in that the image cropping / development processing unit 1184, which performs image cropping and development, and the optical correction / vibration damping processing unit 1185, which performs optical correction processing and vibration damping processing, are located in the display device 1180. Also, due to the relocation of the image cropping / development processing unit to the display device 1180, in this embodiment the overall control CPU 101 is equipped with a recording direction / angle determination unit 30, but not the image cropping / development processing unit 50. An information synthesis unit 1150 is added to the camera body 1101, which synthesizes the recording direction / angle of view information with the main video output from the shooting unit 40. In the display device 1180, the image cropping / development processing unit has been moved as 1184, as in Embodiment 10, an information separation unit 1182 has been added, and a receiving unit 1181, which was omitted from the explanation in Embodiments 1 and 9, as in Embodiment 10, has been added to the figure.

[0758] The order of processing will be explained using the flowchart in Figure 50. For flows that perform the same or similar processing as the flow in Figure 7A, the step number will be increased by 11000 (i.e., the first two digits will be "11"), and the explanation will be omitted. In addition, as an aid to the explanation, in Figure 50, the device shown in Figure 49 is used to perform each step, as indicated to the right of each step. Specifically, steps S11100 to S11700 in Figure 50 are performed by the camera body 1101, and steps S11710 to S11950 are performed by the display device 1180.

[0759] In Example 10, face capture in step S10200 and main shooting in S10400 were performed simultaneously, and then the two captured video data were combined in S10450. In Example 11, face capture was performed in step S11200, and then the recording direction and range were determined in S11400, and the recording direction and range data was output.

[0760] Subsequently, the actual shooting data from S11300, which was executed in parallel, and the recording direction and range data output by S11400 are combined in step S11450.

[0761] While various methods can be considered for combining recording direction and range data, in this embodiment 11, the recording direction and range data are recorded as frame-by-frame metadata for the actual shooting data. This has the same configuration as the metadata in Figure 15.

[0762] The main shooting data created in S11450 is recorded in S11600 and wirelessly transmitted to the display device 11180 in S11700.

[0763] Steps S11710 and later are performed on the display device 1180. In step S11710, the metadata-attached video combined in S11450 is separated again into recording direction / range data and the actual captured video.

[0764] Next, in step S11750, a recording range development process is performed on the main captured video separated in step S11710, using the recording direction and field of view information to extract the video and develop the area within that range.

[0765] In step S11800, optical correction processing is performed to correct optical aberrations in the image developed within the recording range in step S11750. In step S11900, image stabilization processing is performed.

[0766] Of course, in this embodiment as well, the order of steps S11800 and S11900 can be reversed. In other words, the image stabilization process can be performed first, followed by optical correction.

[0767] In step S11950, the display device control unit (video recording means) performs secondary recording, recording the video, which has undergone optical correction processing and vibration damping processing in steps S11800 and S11900, into the large-capacity non-volatile memory 814, and then terminates this process.

[0768] In this embodiment, by sending a composite of the captured video and recording direction / range data as timed metadata in step S11450, the capacity of the video data to be transmitted in step S11700 can be reduced, resulting in reduced power consumption, reduced heat generation, and reduced load on the display device 1180. As in Embodiment 1, gyro data and attitude data obtained from the angular velocity sensor 107 and acceleration sensor 108 may also be transmitted to the display device 1180 during the transfer in step S11700.

[0769] (Example 12) In Example 12, a configuration in which the shooting direction is changed by mechanically driving the direction of the imaging unit will be explained using Figures 51A to 56. Figure 51A is an external view of the camera body 1220 in this embodiment.

[0770] Parts already described in Example 1 are given the same numbers to indicate the same function, and their description in this specification is omitted. The camera body 1220 comprises a shooting / detection unit 1221, connection units 80L and 80R, and a battery unit 90.

[0771] Figure 51B is a perspective view showing details of the imaging / detection unit 1221, which is part of the camera body 1220. The imaging / detection unit 1221 comprises the main body 1210, yaw drive shaft 1201, yaw drive base 1202, pitch drive shaft 1203, and imaging unit 40. The main body 1210 comprises a power switch 11, imaging mode switch 12, face direction detection window 13, start switch 14, stop switch 15, and yaw drive motor 1204.

[0772] The yaw drive motor 1204 drives the yaw drive base 1202 in the yaw direction (left-right direction) via the yaw drive shaft 1201. The yaw drive base 1202 is equipped with a pitch drive motor 1205. The pitch drive motor 1205 drives the imaging unit 40 in the pitch direction (up-down direction) via the pitch drive shaft 1203.

[0773] The imaging unit 40 includes an imaging lens 16 and a solid-state image sensor 42 (not shown). The imaging lens 16 guides light rays from the subject to form an image of the subject on the solid-state image sensor 42.

[0774] Figure 51C is a perspective view showing the imaging unit 40 rotated 30 degrees to the left, and Figure 51D is a perspective view showing the imaging unit 40 facing 30 degrees downwards. As shown in Figure 51C, by driving the yaw drive motor 1204, the yaw drive shaft 1201 and below rotate left and right, making it possible to change the direction of the imaging unit 40 in the yaw direction. As shown in Figure 51D, by driving the pitch drive motor 1205, the pitch drive shaft 1202 and beyond rotate up and down, making it possible to change the direction of the imaging unit 40 in the pitch direction.

[0775] Figure 52 is a functional block diagram of the camera body 1220 according to Embodiment 12. Here, we will use Figure 52 to explain the general flow of processing performed by the camera body 1220. In the following, we will explain the changes from Figure 4, and for the same processing, the same reference numerals will be used and the explanation will be omitted.

[0776] In Figure 52, the camera body 1220 comprises a face direction detection unit 20, an imaging unit drive unit 1230, an imaging unit 40, a developing unit 1250, a primary recording unit 60, a transmission unit 70, and other control units 111. These functional blocks are executed under the control of an overall control CPU 101 (Figure 53) that performs overall control of the camera body 1220.

[0777] The face direction detection unit 20 detects the face direction, estimates the observation direction, and passes this to the imaging unit drive unit 1230. The imaging unit drive unit 1230 performs various calculations based on the observation direction estimated by the face direction detection unit 20 and the outputs of the angular velocity sensor 107 and acceleration sensor 108, and drives the imaging unit 40 to change the imaging direction and imaging angle.

[0778] The shooting unit 40 converts light rays from the subject into an image and passes the image to the developing unit 1250. The developing unit 1250 (developing means) develops the image from the shooting unit 40 and passes the image in the direction the user is looking to the primary recording unit 60. The primary recording unit 60 passes the image to the transmitting unit 70 at the necessary timing. The transmitting unit 70 wirelessly connects to predetermined communication partners, the display device 800 (Figure 1D), the calibrator 850, and the simple display device 900, and communicates with them.

[0779] Figure 53 is a block diagram showing the hardware configuration of the camera body 1220 according to Embodiment 12. Below, only the differences from Figure 5 of Embodiment 1 will be explained. In Figure 53, the camera body 1220 includes a phase detection sensor 1206 and a motor drive circuit 1207. The phase detection sensor 1206 detects the phase of the pitch and yaw of the imaging unit 40 and outputs it to the overall control CPU 101.

[0780] The motor drive circuit 1207 is controlled by the overall control CPU 101 and drives the imaging unit 40 in a desired direction at a desired drive speed.

[0781] The following describes how to use the camera body 1 and the display device 800. Figure 54 is a flowchart outlining the image recording process according to this embodiment, which is performed in the camera body 1220 and the display device 800. For explanatory purposes, Figure 54 indicates on the right side of each step which device shown in Figure 52 is performing that step.

[0782] In step S100, when the power switch 11 is turned ON and power is supplied to the camera body 1, the overall control CPU 101 starts up and reads the startup program from the built-in non-volatile memory 102. After that, the overall control CPU 101 performs preparatory operations to configure the camera body 1 before image capture.

[0783] In step S200, the face direction detection unit 20 detects the face direction and performs a face direction detection process to infer the observation direction. This process is performed at a predetermined frame rate.

[0784] In step S12300, the imaging unit drive unit 1230 performs imaging unit drive processing, which involves calculating the drive amount of the imaging unit 40 and performing drive control. Details of the imaging unit drive processing will be described later with reference to Figure 55.

[0785] In step S400, the imaging unit 40 takes an image and generates image data. In step S12500, the developing unit 1250 performs a developing process on the image data generated in step S400. Details of the developing process will be described later with reference to Figure 56.

[0786] In step S600, the primary recording unit 60 (video recording means) performs a primary recording process in which the video developed in step S12500 is saved as video data in the primary memory 103. In step S700, the transmission unit 70 performs a transfer process to the display device 800 in which the video primary recorded in step S600 is transmitted wirelessly to the display device 800 at a specified timing.

[0787] Steps from step S800 onward are executed by the display device 800. In step S800, the display device control unit 801 performs optical correction processing to correct optical aberrations in the video transferred from the camera body 1 in step S700.

[0788] In step S900, the display device control unit 801 performs image stabilization on the image whose optical aberrations were corrected in step S800. Note that the order of steps S800 and S900 may be reversed. In other words, image stabilization may be performed on the image first, and then optical correction may be performed afterward.

[0789] In step S1000, the display device control unit 801 (video recording means) performs secondary recording, recording the video, which has undergone optical correction processing and vibration damping processing in steps S800 and S900, into the large-capacity non-volatile memory 814, and then terminates this process.

[0790] Figure 55 is a flowchart of the subroutine for step S12300 of the imaging unit drive processing described in Figure 54. In step S12301, the overall control CPU 101 acquires the outputs of the angular velocity sensor 107, the acceleration sensor 108, and the phase detection sensor 1206.

[0791] In step S12302, the overa...

Claims

1. An observation direction detection means is attached to the user's body other than their head and detects the user's observation direction, The aforementioned means is attached to the user's body and captures images, The system includes an image output means that outputs an image corresponding to the observation direction based on the image captured by the imaging means, The imaging device is characterized in that the observation direction detection means comprises an infrared irradiation means that irradiates an infrared irradiation surface including the user's chin with infrared light, and an infrared detection means that detects the reflected light of the infrared light reflected from the infrared irradiation surface, and the position of the user's neck rotation center and chin tip is detected from the reflected light of the infrared light detected by the infrared detection means, and the observation direction is detected from the positions of the neck rotation center and chin tip.

2. The imaging apparatus according to claim 1, characterized in that the observation direction detection means detects the user's observation direction in three dimensions.

3. The imaging apparatus according to claim 1 or 2, characterized in that the observation direction detection means outputs the left-right observation direction of the user's face as the angle of a first detection direction, and outputs the up-down observation direction of the user's face as the angle of a second detection direction perpendicular to the first detection direction.

4. The imaging apparatus according to claim 1, characterized in that the observation direction detection means is located in a distance area that is relatively close to the infrared detection means among a plurality of distance areas of the infrared irradiation surface, and the neck rotation center is set at the left and right center of the distance area and at the closest distance.

5. The imaging apparatus according to any one of claims 1 to 4, characterized in that the observation direction detection means sets the chin position near a distance area with a relatively large distance change among a plurality of distance areas on the infrared irradiation surface, and at the position furthest from the neck rotation center.

6. The observation direction detection means outputs the left-right observation direction of the user's face as the angle of a first detection direction, and outputs the up-down observation direction of the user's face as the angle of a second detection direction perpendicular to the first detection direction. The imaging apparatus according to claim 5, characterized in that the observation direction detection means calculates the movement angle of the chin position with respect to the neck rotation center as the angle of the first detection direction.

7. The observation direction detection means outputs the left-right observation direction of the user's face as the angle of a first detection direction, and outputs the up-down observation direction of the user's face as the angle of a second detection direction perpendicular to the first detection direction. The imaging device according to claim 5 or 6, characterized in that it calculates the angle of the second detection direction based on the light intensity of the reflected light at the chin position.

8. The imaging apparatus according to any one of claims 1 to 7, characterized in that the video output means extracts and outputs an image corresponding to the observation direction from the image captured by the imaging means.

9. The imaging device further comprises calibration means for performing calibration of the observation direction detection means using a wirelessly connected calibrator. The calibrator includes a face detection means that emits infrared light to detect the user's face, The imaging apparatus according to any one of claims 1 to 8, characterized in that the observation direction detection means does not detect the observation direction during the period when the face detection means is emitting the infrared light.

10. The system includes metadata generation means that generates metadata for each frame of video output to the video output means, including video position information indicating the size and position of the video corresponding to the observation direction. The imaging apparatus according to any one of claims 1 to 9, characterized in that the video output means generates a video file in which the metadata and the video of each frame are encoded frame by frame.

11. The imaging means further includes an optical correction value acquisition means for acquiring an optical correction value based on the optical design of the imaging lens, The imaging apparatus according to claim 10, characterized in that the optical correction value is included in the metadata.

12. The system further includes a movement detection means for detecting the movement of the imaging device and acquiring the amount of movement, The imaging apparatus according to claim 10 or 11, characterized in that the amount of movement is included in the metadata.

13. The imaging apparatus according to claim 12, characterized in that the displacement detection means is one of an acceleration sensor for detecting acceleration, an angular velocity sensor for measuring angular velocity, and a magnetic sensor for measuring the direction of a magnetic field.

14. The observation direction detection means is a face direction detection means for detecting the direction of the user's face, During calibration, the acquisition and detection means captures positioning indicators with the imaging means and acquires an image including the positioning indicators, and detects the face direction with the face direction detection means, A position calculation means that calculates the position of the positioning index in the video captured during calibration based on the shape of the positioning index included in the video acquired by the acquisition and detection means, A generation means that generates information showing the relationship between the face direction detected by the acquisition / detection means and the position of the positioning index calculated by the position calculation means, The imaging apparatus according to any one of claims 1 to 13, further comprising: a first calibration means that performs calibration of the center position of the target field of view according to the face direction detected by the face direction detection means, based on the information generated by the generation means.

15. A movement amount detection means for detecting the amount of movement of the imaging device during video capture by the imaging means, and an observation direction correction means for correcting the amount of movement in the observation direction, The observation direction correction means is If the movement amount detection means detects that the imaging device is accelerating, the movement amount in the observation direction is delayed. The imaging device according to any one of claims 1 to 14, characterized in that, when the movement amount detection means detects that the imaging device is decelerating, the movement in the observation direction is accelerated to cover the delayed amount.

16. The imaging apparatus according to claim 15, characterized in that the movement amount detection means detects the movement amount by comparing images of multiple frames obtained by video imaging by the imaging means.

17. The imaging apparatus according to claim 15 or 16, characterized in that the video output means outputs a portion of the video captured by the imaging means, which is cropped according to the observation direction.

18. A driving means for driving the imaging direction of the imaging means in the yaw direction and the pitch direction, The imaging apparatus according to any one of claims 15 to 17, further comprising an imaging direction changing means that changes the imaging direction of the imaging means by the driving means according to the observation direction.

19. The imaging apparatus according to any one of claims 15 to 18, characterized in that the observation direction correction means corrects the movement speed in the observation direction in the video output output by the video output means so that it is substantially constant.

20. The imaging apparatus according to any one of claims 1 to 19, characterized in that the detection optical axis of the observation direction detection means and the imaging optical axis of the imaging means are arranged to face in different directions from each other.

21. The imaging apparatus according to claim 20, characterized in that the detection optical axis of the observation direction detection means is positioned to point from the observation direction detection means toward the user's jaw.

22. The imaging apparatus according to claim 20 or 21, characterized in that the imaging optical axis of the imaging means is arranged to face the user from the imaging means.

23. The imaging device, in which the observation direction detection means and the imaging means are integrally configured, is incorporated into the camera body. The imaging device according to any one of claims 20 to 22, characterized in that, when the user is wearing the camera body, the horizontal length is longer than the vertical length when viewed from the front of the user.

24. The camera body is equipped with a fixing means that makes contact with the user's body or clothing. The imaging device according to claim 23, characterized in that the fixing means is positioned near the left and right ends of the imaging device when the user is wearing the camera body.

25. The imaging device according to claim 24, wherein the camera body further comprises a grounding angle adjustment means for adjusting the angle at which the fixing means makes contact with the user.

26. The imaging device is connected to a neck strap for the user to wear the imaging device around their neck. The imaging device according to claim 23, characterized in that both ends of the neck-hanging means are connected near the imaging device and its left and right ends when the user is wearing the camera body.

27. The imaging apparatus according to claim 26, wherein the neck-hanging means is provided with a neck-hanging angle adjustment means for adjusting the angle of the neck-hanging means with respect to the imaging apparatus.

28. The neck-hanging means has a cross-sectional shape that is not perfectly circular. The imaging device according to claim 26, characterized in that, when the user is wearing the camera body, the distance between the parts of the neck-hanging means that are symmetrically positioned with respect to the imaging device decreases as you move from the bottom to the top.

29. The imaging device is connected to the power supply by the neck-hanging means, The imaging device according to any one of claims 26 to 28, characterized in that it is positioned behind the user's neck when the user is wearing the camera body.

30. The imaging device is connected by the power supply means and the power supply means, The imaging apparatus according to claim 29, characterized in that the power supply means is arranged to pass through the inside of the neck-hanging means.

31. An observation direction detection step in which the user's observation direction is detected by an observation direction detection means attached to the user's body other than the user's head, The imaging step involves capturing an image using an imaging device attached to the user's body, The imaging step includes a video output step that outputs an image corresponding to the observation direction based on the image captured in the imaging step, The method for controlling an imaging device is characterized in that the observation direction detection step comprises an infrared irradiation step of irradiating an infrared irradiation surface including the user's chin with infrared light, and an infrared detection step of detecting the reflected infrared light reflected from the infrared irradiation surface, the position of the user's neck rotation center and chin tip detected from the reflected infrared light detected in the infrared detection step, and the observation direction detected from the positions of the neck rotation center and chin tip.

Images