Display device and control method for display device

Abstract

To generate cut-out moving images cut out at a desired position with a simple operation.SOLUTION: A display unit has: acquisition means that acquires moving image data obtained by adding, to moving images, information on a cut-out area of the moving images set based on a face direction of a user who photographs the moving images; display control means that displays a frame indicating the cut-out area during playback of the moving image data; operation means that receives an operation to the frame for correcting the cut-out area; correction means that adds correction information on the cut-out area corrected by the operation to the moving image data; and creation means that creates cut-out moving images from the moving images based on the information on the cut-out area and the correction information.SELECTED DRAWING: Figure 59A

Description

[Technical Field]

[0001] The present invention relates to a display device and a method for controlling a display device. [Background technology]

[0002] Traditionally, taking images with a camera requires the photographer to keep the camera pointed in the direction they want to capture the image. This means that the photographer's hands are tied to the act of taking the image, preventing them from doing anything else, or concentrating their attention on the act of taking the image prevents them from focusing on the experience of being in the moment.

[0003] For example, in terms of image capture operations, a parent who is taking the picture cannot play with their child while capturing images, and conversely, if they try to play with their child, they cannot capture images, which presents a challenge.

[0004] Furthermore, in terms of focusing attention on imaging, when imaging during a sporting event, the photographer may not be able to cheer or remember the details of the game, and focusing attention on watching the sport may prevent them from imaging. Similarly, when imaging during a group trip, the photographer may not be able to experience the emotions at the same level as the other members, and prioritizing the experience may lead to neglecting imaging.

[0005] One way to address these challenges is to use a head-mounting accessory to fix the action camera to the head and capture images in the direction of observation, allowing the photographer to capture images without being tied down by the camera's hands. Another method is to use a 360-degree camera to capture a wide area, allowing the participant to focus on the experience during the experience, and then, after the experience is over, to cut out and edit the necessary footage from the captured 360-degree video to preserve the video of the experience.

[0006] However, the former method requires the cumbersome act of attaching a head-mounting accessory to the head, to which the action camera 901 body is fixed, as shown in Figure 47(a). Furthermore, as shown in Figure 47(b), when the photographer attaches the action camera 901 to their head using the head-mounting accessory 902, it looks unsightly and causes problems such as messing up the photographer's hairstyle. In addition, the photographer was bothered by the weight and presence of the head-mounting accessory 902 and the action camera 901 attached to their head, and was also concerned about the unsightly appearance to third parties. As a result, in the state shown in Figure 47(b), the photographer was unable to concentrate on the experience, or felt resistance to being in the state shown in Figure 47(b), making it difficult to take images.

[0007] On the other hand, the latter method requires a series of operations such as image conversion and specifying the cropping position. For example, a 360-degree camera 903 equipped with a lens 904 and a shooting button 905 is known, as shown in Figure 48. The lens 904 is one of a pair of hemispherical fisheye lenses configured on both sides of the housing of the 360-degree camera 903, and the 360-degree camera 903 performs 360-degree photography using this pair of fisheye lenses. In other words, 360-degree photography is performed by combining the projected images from this pair of fisheye lenses.

[0008] Figure 49 shows an example of the conversion process for images captured by the 360-degree camera 903.

[0009] Figure 49(a) is an example of an image obtained by 360-degree imaging with the 360-degree camera 903, and includes the subjects: the photographer 906, the child 907, and the tree 908. This image is a hemispherical optical system image obtained by combining the projection images of a pair of fisheye lenses, therefore, the image is captured Person 906 is greatly distorted. Furthermore, the child 907, the subject that photographer 906 was trying to photograph, had its body positioned at the periphery of the hemispherical optical system, causing its body to be greatly distorted and stretched from side to side. On the other hand, the tree 908 was positioned directly in front of lens 904, and therefore was photographed without significant distortion.

[0010] To create an image representing the field of view that a person normally sees from the image in Figure 49(a), it is necessary to cut out a portion of it, transform it into a plane, and display it.

[0011] Figure 49(b) is an image cropped from the image in Figure 49(a) that is positioned directly in front of lens 904. In the image in Figure 49(b), the tree 908 is in the center, representing a field of view similar to that of a normal human. However, the child 907 that the photographer 906 was trying to capture is not included in Figure 49(b), so the cropping position must be changed. Specifically, the cropping position in Figure 49(a) must be to the left of the tree 908 and 30° downward from the perspective of the drawing. After this cropping work is performed, the image is transformed into a plane and displayed as Figure 49(c). Thus, in order to obtain the image in Figure 49(c) that the photographer was trying to capture from the image in Figure 49(a), it is necessary to crop the required area and transform it into a plane (hereinafter referred to as "trimming"). Therefore, although the photographer can concentrate on the experience (while capturing images), the amount of work that follows becomes enormous.

[0012] Patent Document 1 discloses a technique for correcting the image cropping position in which the user changes the shake correction strength during image playback, and the image cropping size and position are determined based on the shake correction strength. [Prior art documents] [Patent Documents]

[0013] [Patent Document 1] Japanese Patent Publication No. 2017-212550 [Overview of the Initiative] [Problems that the invention aims to solve]

[0014] Even though the cropping area can be indirectly corrected by changing the shake correction strength, it is difficult for users to correct it to the intended position without directly changing the cropping area.

[0015] Therefore, the present invention aims to provide a technology for generating a cropped video, cut out at a desired position, with simple operation. [Means for solving the problem]

[0016] The display device according to the present invention is Acquisition means for acquiring video data in which information about the video's cropping region, which is set based on the face orientation of the user shooting the video, is added to the video. A display control means that displays a frame indicating the cropped area when the aforementioned video data is played back, An operation that accepts an operation on the frame for correcting the aforementioned cutout area. component and, The cropped area corrected by the above operation Shows the correction amount. Correction means for adding correction information to the video data, A generation means that generates a cut video from the video based on the information of the cut region and the correction information. to have death, The correction information includes at least one of the following: a cropping position correction amount, which is the amount of correction for the position of the cropped area corrected by user operation, and a cropping size correction amount, which is the amount of correction for the size of the cropped area corrected by user operation. The generation means uses the correction information for the correction frame, which is the frame that received the operation, to correct the cutout region of the unoperated frame, which is the frame that did not receive the operation, and generates the cutout video. do It is characterized by the following: [Effects of the Invention]

[0017] According to the present invention, a cropped video extracted at a desired position can be generated with simple operations. [Brief explanation of the drawing]

[0018] [Figure 1A] This is an external view of the camera body, including the imaging and detection unit, as an imaging device according to Example 1. [Figure 1B] This diagram shows the camera body being worn by the user. [Figure 1C] This is a view of the battery section of the camera body from the rear, as shown in Figure 1A. [Figure 1D] This is an external view of a display device as a portable device according to Embodiment 1, which is composed of a separate component from the camera body. [Figure 2A] This is a front view of the imaging and detection unit. [Figure 2B] This diagram shows the shape of the band portion of the connection part on the camera body. [Figure 2C] This is a view of the imaging and detection unit from the back. [Figure 2D] This is a top view of the imaging and detection unit. [Figure 2E] This diagram shows the configuration of an infrared detection processing device located inside the shooting and detection unit, below the face direction detection window in the camera body. [Figure 2F] This is a view from the left side of the user, showing the camera body as it is worn by the user. [Figure 3] This is a diagram illustrating the details of the battery section. [Figure 4] This is a functional block diagram of the camera body according to Example 1. [Figure 5] This is a block diagram showing the hardware configuration of the camera body. [Figure 6] Block diagram showing the hardware configuration of a display device. [Figure 7A] This flowchart shows an overview of the image recording process according to Example 1, which is performed in the camera body and display device. [Figure 7B] This is a flowchart of the subroutine for the preparation operation process in step S100 of Figure 7A, according to Example 1. [Figure 7C] This is a flowchart of the subroutine for the face direction detection process in step S200 of Figure 7A, according to Example 1. [Figure 7D] This is a flowchart of the subroutine for determining the recording direction and range in step S300 of Figure 7A, according to Example 1. [Figure 7E]This is a flowchart of the subroutine for the recording range development process in step S500 of Figure 7A, according to Example 1. [Figure 7F] This diagram illustrates the process from steps S200 to S500 in Figure 7A in video mode. [Figure 8A] This diagram shows the user's image as seen through the face direction detection window. [Figure 8B] This diagram shows the case where a fluorescent light in the room is reflected as a background in the image of the user as seen through the face direction detection window. [Figure 8C] Figure 8B shows an image of the user and the fluorescent lamp in the background, when the infrared LED of the infrared detection processing device is not turned on, and the image is formed by the sensor of the infrared detection processing device through the face direction detection window. [Figure 8D] Figure 8B shows an image of the user and the fluorescent lamp in the background, when the infrared LED is turned on and the image is formed by the sensor of the infrared detection processing device through the face direction detection window. [Figure 8E] This figure shows the difference image calculated from the images in Figures 8C and 8D. [Figure 8F] This figure shows the case where the contrast of the difference image in Figure 8E is adjusted to match the light intensity of the infrared reflected light projected onto the user's face and neck. [Figure 8G] Figure 8F is a diagram in which symbols indicating different parts of the user's body, as well as double circles and black circles indicating the neck and chin positions, are superimposed. [Figure 8H] This figure shows the difference image calculated using the same method as in Figure 8E, when the user's face is turned to the right. [Figure 8I] Figure 8H shows the double circles and black circles superimposed to indicate the positions of the neck and chin. [Figure 8J] This diagram shows the user's image as seen through the face direction detection window when the user's face is turned 33° upward from the horizontal. [Figure 8K] This figure shows the difference image calculated using the same method as in Figure 8E, with double circles and black circles indicating the neck position and chin position superimposed on the image when the user's face is turned 33° above the horizontal. [Figure 9] This is a timing chart showing the timing of when the infrared LEDs light up. [Figure 10] This diagram illustrates the vertical movement of the user's face. [Figure 11A] This diagram shows the target field of view in an ultra-wide-angle image captured by the camera's shooting unit when the user is facing forward. [Figure 11B] This figure shows the image of the target field of view in Figure 11A, extracted from an ultra-wide-angle image. [Figure 11C] This diagram shows the target field of view in an ultra-wide-angle image when the user is observing subject A. [Figure 11D] This figure shows the image of the target field of view, extracted from ultra-wide-angle footage in Figure 11C, with distortion and shaking corrected. [Figure 11E] This figure shows the target field of view in ultra-wide-angle video when the user is observing subject A with a field of view setting smaller than that shown in Figure 11C. [Figure 11F] This figure shows the image of the target field of view, extracted from ultra-wide-angle footage in Figure 11E, with distortion and shaking corrected. [Figure 12A] This figure shows an example of the target field of view in ultra-wide-angle video. [Figure 12B] This figure shows an example of a target field of view in ultra-wide-angle video, where the field of view setting is the same as the target field of view in Figure 12A, but the observation direction is different. [Figure 12C] This figure shows another example of a target field of view in ultra-wide-angle video, with the same field of view setting as the target field of view in Figure 12A, but with a different observation direction. [Figure 12D] This figure shows an example of a target field of view in ultra-wide-angle video, where the observation direction is the same as the target field of view in Figure 12C, but the field of view setting value is smaller. [Figure 12E] This figure shows an example where a reserve area is added around the target field of view, as shown in Figure 12A. [Figure 12F] This figure shows an example where a reserve area with the same vibration isolation level as the reserve area in Figure 12E is added around the target field of view shown in Figure 12B. [Figure 12G] This figure shows an example where a reserve area with the same vibration isolation level as the reserve area in Figure 12E is added around the target field of view shown in Figure 12D. [Figure 13] This diagram shows the menu screen for various video mode settings, which is displayed on the display unit of the camera before image capture. [Figure 14] Figure 7A is a flowchart of the subroutine for the primary recording process in step S600. [Figure 15] This diagram shows the data structure of the video file generated by the primary recording process. [Figure 16] Figure 7A is a flowchart of the subroutine for the transfer process to the display device in step S700. [Figure 17] Figure 7A is a flowchart of the subroutine for the optical correction process in step S800. [Figure 18] This figure illustrates the case where distortion correction is performed in step S803 of Figure 17. [Figure 19] Figure 7A is a flowchart of the vibration isolation subroutine in step S900. [Figure 20] This figure shows the details of the calibrator used in the calibration process according to Example 2. [Figure 21] This is a flowchart of the calibration process according to Embodiment 2, which is performed in the camera body and calibrator. [Figure 22A] This figure shows the screen displayed on the calibrator's display unit during the calibration operation in the direction facing the user, in step S3103 of Figure 21. [Figure 22B] This is a perspective view showing the user holding the calibrator forward in accordance with the instructions shown in the instruction display in Figure 22A. [Figure 22C] This is a schematic diagram showing the entire ultra-wide-angle image captured by the imaging lens in the state shown in Figure 22B. [Figure 22D] Figure 22C is a schematic diagram showing an image with aberrations corrected from the ultra-wide-angle image shown. [Figure 22E] This is a schematic diagram showing the face direction image recorded by the face direction detection unit in step S3108 of Figure 21 during the calibration operation for the user's frontal direction. [Figure 22F] This is a schematic diagram illustrating the determination method in step S3107 of Figure 21. [Figure 23A] This figure shows the screen displayed on the calibrator's display unit during the user's upward right-hand calibration operation in step S3103 of Figure 21. [Figure 23B] This is a perspective view showing the user holding the calibrator to the upper right in accordance with the instructions shown in the instruction display in Figure 23A. [Figure 23C] This is a schematic diagram showing the entire ultra-wide-angle image captured by the imaging lens in the state shown in Figure 23B. [Figure 23D] Figure 23C is a schematic diagram showing an image with aberrations corrected from the ultra-wide-angle image shown. [Figure 23E] This is a schematic diagram showing the face direction image recorded by the face direction detection unit in step S3108 of Figure 21 during the calibration operation of the user's right hand in the upward direction. [Figure 24] This is a diagram illustrating the delayed image extraction in Example 3. [Figure 25] This figure shows the trajectory of the face held in Example 3. [Figure 26] This is a flowchart of the motion sickness prevention process according to Example 3. [Figure 27] This is a conceptual diagram of the cropping range correction process according to Example 4. [Figure 28] This is a flowchart of the cropping range correction process according to Example 4. [Figure 29] This is a schematic diagram illustrating the relationship between the user's field of view and the target field of view when observing a close-range subject in Example 1. [Figure 30] This is an external view of the camera body used as an imaging device according to Example 5. [Figure 31] This is a block diagram showing the hardware configuration of the camera body according to Example 5. [Figure 32] This is a schematic diagram illustrating the relationship between the user, calibrator, and target field of view during calibration, including parallax correction processing, in Example 5. [Figure 33A] This is a flowchart of the parallax correction mode processing, which is part of the preparation process in step S100 of Figure 7A in Example 5. [Figure 33B] This is a flowchart of the recording direction and range determination subroutine for S300, as explained in Figure 7A of Example 5. [Figure 34] This is a schematic diagram showing the relationship between the defocus map created in step S5302 of Figure 33B and the recording direction. [Figure 35] This is a flowchart of the observation direction determination process according to Example 6. [Figure 36A] This figure shows the relationship between the user's observation direction detection state for each frame and the captured image, according to Example 6. [Figure 36B] This figure shows the relationship between the user's observation direction detection state for each frame and the captured image in the subject loss mode according to Example 6. [Figure 37] This is a diagram illustrating the relationship between the observation direction and the face region that can be used to detect the face direction, according to Example 7. [Figure 38] This is a flowchart of the observation direction determination process when acquiring face direction according to Example 7, which is performed instead of the process in step S6004 in Figure 35. [Figure 39] This figure shows the relationship between face direction and face direction reliability in Example 7. [Figure 40] This is a conceptual diagram of the observation direction determination process when acquiring face direction in Example 7. [Figure 41] This is an enlarged view showing the imaging and detection unit from the side. [Figure 42] This is a side view showing the camera body attached to the user. [Figure 43] This is an enlarged view showing the shooting / detection unit from the side, with the connection part hidden. [Figure 44]This is a side view showing how the camera body looks when the connection part is hidden and the user has attached it. [Figure 45] This figure shows the band portion and the connection surface, which is the cut surface of the electrical cable that is integrally formed with it. [Figure 46A] This is a block diagram showing the hardware configuration of a display device connected to a camera body, which serves as an imaging device according to Example 9. [Figure 46B] This is a functional block diagram of the camera body according to Example 9. [Figure 47] This figure shows an example of a camera configuration that is fixed to the head using a conventional head-fixing accessory. [Figure 48] This figure shows an example configuration of a conventional 360-degree camera. [Figure 49] This figure shows an example of the conversion process for images captured by a 360-degree camera (Figure 48). [Figure 50] This is an example diagram showing a portion of a video clip. [Figure 51] This is a diagram explaining the imaging modes of the camera body. [Figure 52A] This flowchart shows an overview of the shooting recording process according to Example 10. [Figure 52B] This is a flowchart of the preparation process. [Figure 52C] This is a flowchart for the full-area development process. [Figure 52D] This is a flowchart of the primary recording process. [Figure 53] This figure shows the data structure of the video file relating to Example 10. [Figure 54] This diagram illustrates the correction of the cropping area in a display device. [Figure 55] This diagram illustrates a full-range video. [Figure 56] This figure shows an example of a user dragging the cropping area frame. [Figure 57] This diagram illustrates the correction of the extracted region of the unprocessed frame. [Figure 58]This diagram illustrates a sample video generated by a display device. [Figure 59A] This is a flowchart of the video editing process according to Example 10. [Figure 59B] This is a flowchart for the process of acquiring correction information for unprocessed frames. [Figure 59C] This is a flowchart for the process of generating extracted video clips. [Figure 60] This diagram illustrates the correction of the cropped area during playback. [Figure 61A] This is a flowchart of the video editing process according to Example 11. [Figure 61B] This is a flowchart for the process of acquiring correction information for unprocessed frames. [Modes for carrying out the invention]

[0019] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0020] (Example 1) Figures 1A to 1D illustrate a camera system according to this embodiment, consisting of a camera body 1 including an imaging / detection unit 10 and a display device 800 which is configured separately. In this embodiment, the camera body 1 and the display device 800 are shown as separate components, but they may be configured as an integrated unit. The user who wears the camera body 1 around their neck will be referred to as the user below.

[0021] Figure 1A is an external view of the camera body 1.

[0022] In Figure 1A, the camera body 1 includes a shooting / detection unit 10, a battery unit 90, and a connection unit 80 that connects the shooting / detection unit 10 and the battery unit 90 (power supply means).

[0023] The imaging and detection unit 10 includes a face direction detection window 13, a start switch 14, a stop switch 15, an imaging lens 16, an LED 17, and microphones 19L and 19R.

[0024] The face direction detection window 13 transmits infrared light and its reflected light emitted from the infrared LED lighting circuit 21 (Figure 5: infrared irradiation means), which is built into the shooting / detection unit 10 and is used to detect the position of various parts of the user's face.

[0025] The start switch 14 is a switch used to start imaging.

[0026] The stop switch 15 is a switch used to stop image acquisition.

[0027] The imaging lens 16 guides the light rays to be imaged to the solid-state image sensor 42 (Figure 5) inside the imaging / detection unit 10.

[0028] LED17 is an LED that indicates imaging is in progress or displays a warning.

[0029] Microphones 19R and 19L are microphones that pick up ambient sounds. Microphone 19L picks up sounds from the left side of the user's surroundings (right side in Figure 1A), and microphone 19R picks up sounds from the right side of the user's surroundings (left side in Figure 1A).

[0030] Figure 1B shows the camera body 1 being worn by the user.

[0031] When the battery unit 90 is attached to the user's back and the imaging / detection unit 10 is attached to the user's front, the imaging / detection unit 10 is biased and supported towards the chest by the connecting parts 80, which are connected to the left and right ends of the imaging / detection unit 10. As a result, the imaging / detection unit 10 is positioned in front of the user's collarbone. At this time, the face direction detection window 13 is positioned below the user's chin. Inside the face direction detection window 13 is an infrared focusing lens 26, which will be shown later in Figure 2E. The optical axis of the imaging lens 16 (imaging optical axis) and the optical axis of the infrared focusing lens 26 (detection optical axis) are oriented in different directions, and the face direction detection unit 20 (face direction detection means), which will be described later, detects the user's observation direction from the position of each part of the face. This enables imaging in that observation direction by the imaging unit 40 (imaging means), which will be described later.

[0032] Methods for adjusting the setting position due to individual differences in body shape and clothing will be discussed later.

[0033] Furthermore, by positioning the shooting / detection unit 10 on the front of the body and the battery unit 90 on the back, the weight is distributed, which reduces user fatigue and suppresses displacement caused by centrifugal force when the user moves.

[0034] In this embodiment, the image capture / detection unit 10 is shown to be mounted in a position near the user's collarbone, but this is not the only option. That is, as long as the camera body 1 can detect the user's viewing direction using the face direction detection unit 20 and capture images in that viewing direction using the image capture unit 40, the camera body 1 may be mounted anywhere on the user's body other than their head.

[0035] Figure 1C is a view of the battery unit 90 from the rear of Figure 1A.

[0036] In Figure 1C, the battery unit 90 includes a charging cable insertion port 91, adjustment buttons 92L and 92R, and a spine protection cutout 93.

[0037] The charging cable port 91 is an insertion port for a charging cable (not shown), and through this charging cable, the internal battery 94 is charged from an external power source, or power is supplied to the shooting / detection unit 10.

[0038] The adjustment buttons 92L and 92R are used to adjust the length of the band sections 82L and 82R of the connection section 80. These are buttons. Adjustment button 92L is for adjusting the band section 82L on the left side, and adjustment button 92R is for adjusting the band section 82R on the right side. In this embodiment, the lengths of the band sections 82L and 82R are adjusted independently using adjustment buttons 92L and 92R, but it is also possible to adjust the lengths of the band sections 82L and 82R simultaneously with a single button. Hereinafter, the band sections 82L and 82R will be collectively referred to as band section 82.

[0039] The spine-relief cutout 93 is a cutout designed to avoid contact with the user's spine, preventing the battery unit 90 from touching the spine. By avoiding the protruding part of the human spine, it reduces discomfort during wear and prevents the device from moving from side to side during use.

[0040] Figure 1D is an external view of the display device 800 as a portable device according to Embodiment 1, which is configured separately from the camera body 1.

[0041] In Figure 1D, the display device 800 includes button A802, display unit 803, button B804, front camera 805, face sensor 806, angular velocity sensor 807, and acceleration sensor 808. Although not shown in Figure 1D, it also includes a wireless LAN capable of high-speed connection with the camera body 1.

[0042] Button A802 is a button that functions as the power button for the display device 800. It accepts power ON and OFF operations by long-pressing and other processing timing instructions by short-pressing.

[0043] The display unit 803 allows users to view images captured by the camera body 1 and display menu screens necessary for settings. In this embodiment, a transparent touch sensor is also provided on the top surface of the display unit 803 to accept touch operations on the displayed screen (e.g., the menu screen).

[0044] Button B804 functions as a calibration button 854, which is used in the calibration process described later.

[0045] The front camera 805 is a camera capable of capturing images of a person observing the display device 800.

[0046] The face sensor 806 detects the face shape and viewing direction of a person observing the display device 800. The specific structure of the face sensor 806 is not particularly limited, but it can be implemented using various sensors such as a structural light sensor, a ToF sensor, or a millimeter-wave radar.

[0047] The angular velocity sensor 807 is located inside the display device 800 and is therefore shown with a dotted line in the perspective view. The display device 800 in this embodiment also has a calibrator function, which will be described later, and is therefore equipped with a gyro sensor in three directions: X, Y, and Z.

[0048] The accelerometer 808 detects the orientation of the display device 800.

[0049] In this embodiment, a standard smartphone is used as the display device 800, and the camera system according to the present invention can be implemented by making the firmware of the smartphone compatible with the firmware of the camera body 1. However, the camera system according to the present invention can also be implemented by making the firmware of the camera body 1 compatible with the application and OS of the smartphone used as the display device 800.

[0050] Figures 2A to 2F are diagrams illustrating the imaging and detection unit 10 in detail. The same number is used to indicate the same function for the parts described above, and the explanation in this specification is omitted.

[0051] Figure 2A is a front view of the imaging and detection unit 10.

[0052] The connection section 80 connects to the imaging / detection unit 10 at a right-side connection section 80R located on the right side of the user's body (left side in Figure 2A) and a left-side connection section 80L located on the left side of the user's body (right side in Figure 2A). More specifically, the connection section 80 is divided into a rigid angle-holding section 81 and a band section 82 that maintain the angle with respect to the imaging / detection unit 10. That is, the right-side connection section 80R has an angle-holding section 81R and a band section 82R, and the left-side connection section 80L has an angle-holding section 81L and a band section 82L.

[0053] Figure 2B shows the shape of the band portion 82 of the connecting portion 80. In this figure, the angle holding portion 81 is shown transparently to illustrate the shape of the band portion 82.

[0054] The band portion 82 includes a connection surface 83 and an electrical cable 84.

[0055] The connection surface 83 is the connection surface between the angle holding part 81 and the band part 82, and has a cross-sectional shape that is not a perfect circle, in this case an ellipse. Hereinafter, among the connection surfaces 83, the connection surfaces 83 that are positioned symmetrically on the right side (left side in Figure 2B) and left side (right side in Figure 2B) of the user's body when the camera body 1 is attached will be referred to as the right connection surface 83R and the left connection surface 83L. The right connection surface 83R and the left connection surface 83L are shaped like the Japanese katakana character "ハ". That is, the distance between the right connection surface 83R and the left connection surface 83L decreases as you move from the bottom to the top in Figure 2B. As a result, when the user puts the camera body 1 on, the long axis direction of the connection surface 83 of the connection part 80 is aligned with the user's body, resulting in a comfortable fit when the band part 82 touches the user's body and preventing the shooting / detection unit 10 from moving in the left, right, front, or back directions.

[0056] The electrical cable 84 (power supply means) is wired inside the band section 82L and is a cable that electrically connects the battery section 90 and the imaging / detection section 10. The electrical cable 84 connects the power supply of the battery section 90 to the imaging / detection section 10 and also transmits and receives electrical signals to and from the outside.

[0057] Figure 2C shows the imaging / detection unit 10 viewed from the back. Since Figure 2C is a view from the side that contacts the user's body, i.e., the opposite side of Figure 2A, the positional relationship between the right connection part 80R and the left connection part 80L is reversed compared to Figure 2A.

[0058] The imaging and detection unit 10 is equipped with a power switch 11, an imaging mode switch 12, and a chest connection pad 18 on its back side.

[0059] The power switch 11 is a power switch that switches the power of the camera body 1 ON / OFF. In this embodiment, the power switch 11 is a slide lever type switch, but is not limited to this. For example, the power switch 11 may be a push type switch, or it may be a switch integrated with a slide cover (not shown) of the imaging lens 16.

[0060] The imaging mode switch 12 (changing means) is a switch that changes the imaging mode and can change the mode related to imaging. In this embodiment, the imaging mode switch 12 can switch to still image mode, video mode, and a pre-setting mode set using the display device 800, which will be described later. In this embodiment, the imaging mode switch 12 selects one of "Photo," "Normal," or "Pri" as shown in Figure 2C by sliding the lever. This is a switch in the form of a sliding lever. The imaging mode is changed by sliding to "Photo" to switch to still image mode, by sliding to "Normal" to switch to video mode, and by sliding to "Pri" to switch to pre-setting mode. Note that the imaging mode switch 12 is not limited to the form of this embodiment, as long as it is a switch that can change the imaging mode. For example, the imaging mode switch 12 may consist of three buttons: "Photo", "Normal", and "Pri".

[0061] The chest connection pads 18 (fixing means) are the parts that come into contact with the user's body when the imaging / detection unit 10 is biased against the user's body. As shown in Figure 2A, the imaging / detection unit 10 is shaped so that its horizontal (left-right) length is longer than its vertical (up-down) length when attached, and the chest connection pads 18 are positioned near the left and right ends of the imaging / detection unit 10. This arrangement makes it possible to suppress lateral rotational blur during imaging with the camera body 1. In addition, the presence of the chest connection pads 18 prevents the power switch 11 and the imaging mode switch 12 from coming into contact with the body. Furthermore, the chest connection pads 18 also serve to prevent heat from being transferred to the user's body even if the temperature of the imaging / detection unit 10 rises during long-term imaging, and also play a role in adjusting the angle of the imaging / detection unit 10.

[0062] Figure 2D is a top view of the imaging and detection unit 10.

[0063] As shown in Figure 2D, a face direction detection window 13 is provided in the center of the upper surface of the imaging / detection unit 10, and the chest connection pad 18 protrudes from the imaging / detection unit 10.

[0064] Figure 2E shows the configuration of the infrared detection processing device 27, which is located inside the imaging and detection unit 10 and positioned below the face direction detection window 13.

[0065] The infrared detection and processing device 27 includes an infrared LED 22 and an infrared focusing lens 26.

[0066] The infrared LED 22 emits infrared light 23 (Figure 5) towards the user.

[0067] The infrared focusing lens 26 is a lens that focuses the reflected light rays 25 (Figure 5) reflected from the user when infrared rays 23 are emitted from the infrared LED 22 onto a sensor (not shown) of the infrared detection processing device 27.

[0068] Figure 2F shows the camera body 1 as seen from the left side of the user's body while the user is wearing it.

[0069] The angle adjustment button 85L is a button located on the angle holding unit 81L and is used to adjust the angle of the shooting / detection unit 10. Although not shown in this figure, an angle adjustment button 85R is also set inside the angle holding unit 81R on the opposite side, in a position symmetrical to the angle adjustment button 85L. Hereafter, when referring to the angle adjustment buttons 85R and 85L collectively, they will be referred to as the angle adjustment button 85.

[0070] The angle adjustment button 85 is visible in Figures 2A, 2C, and 2D, but it has been omitted for the sake of simplicity in this explanation.

[0071] The user can change the angle between the imaging / detection unit 10 and the angle holding unit 81 by pressing the angle adjustment button 85 and moving the angle holding unit 81 up and down in the direction of Figure 2F. In addition, the chest connection pad 18 can have its protrusion angle changed. Through the action of these two angle-changing members (angle adjustment button 85 and chest connection pad 18), the imaging / detection unit 10 can adjust the orientation of the imaging lens 16 to be horizontal to accommodate individual differences in the shape of the user's chest position.

[0072] Figure 3 is a diagram illustrating the details of the battery unit 90.

[0073] Figure 3(a) is a view of the battery unit 90 from the rear, with a partial perspective.

[0074] As shown in Figure 3(a), the battery unit 90 has two batteries, a left battery 94L and a right battery 94R (hereinafter also collectively referred to as battery 94), symmetrically mounted inside to balance its weight. By symmetrically arranging the batteries 94 in the center of the battery unit 90 in this way, the weight balance between the left and right sides is adjusted, preventing the camera body 1 from shifting position. The battery unit 90 may also be configured to house only one battery.

[0075] Figure 3(b) is a view of the battery unit 90 from above. In this figure as well, the battery 94 is shown in perspective.

[0076] As shown in Figure 3(b), the relationship between the spine-guard cutout 93 and the battery 94 can be seen. By symmetrically arranging the battery 94 on both sides of the spine-guard cutout 93 in this way, it is possible to attach the relatively heavy battery unit 90 to the user without causing any burden.

[0077] Figure 3(c) is a view of the battery unit 90 from the back. Figure 3(c) is a view from the side that comes into contact with the user's body, that is, the opposite side from Figure 3(a).

[0078] As shown in Figure 3(c), the spine-guard cutout 93 is located in the center along the user's spine.

[0079] Figure 4 is a functional block diagram of the camera body 1. Details will be explained later, but here we will use Figure 4 to describe the general flow of processing performed by the camera body 1.

[0080] In Figure 4, the camera body 1 comprises a face direction detection unit 20, a recording direction / angle determination unit 30, a shooting unit 40, an image cropping / development processing unit 50, a primary recording unit 60, a transmission unit 70, and other control units 111. These functional blocks are executed under the control of the overall control CPU 101 (Figure 5), which controls the entire camera body 1.

[0081] The face direction detection unit 20 (observation direction detection means) is a functional block executed by the infrared LED 22 and infrared detection processing device 27 mentioned earlier, which detects the face direction, infers the observation direction, and passes this to the recording direction / angle determination unit 30.

[0082] The recording direction / angle determination unit 30 (recording direction determination means) performs various calculations based on the observation direction inferred by the face direction detection unit 20 to determine the position and range information for extracting the image from the shooting unit 40, and passes this information to the image extraction / development processing unit 50.

[0083] The shooting unit 40 converts light rays from the subject into an image and passes that image to the image extraction and development processing unit 50.

[0084] The image extraction and development processing unit 50 (development means) uses information from the recording direction and field of view determination unit 30 to extract and develop the video from the shooting unit 40, thereby passing only the video in the direction the user is looking to the primary recording unit 60.

[0085] The primary recording unit 60 is a functional block composed of the primary memory 103 (Figure 5), etc. The system records video information and transmits it to the transmission unit 70 at the appropriate time.

[0086] The transmitting unit 70 (video output means) wirelessly connects to predetermined communication partners, namely the display device 800 (Figure 1D), the calibrator 850, and the simple display device 900, and communicates with them.

[0087] The display device 800 is a display device that can connect to the transmission unit 70 via a high-speed wireless LAN (hereinafter referred to as "high-speed wireless"). In this embodiment, wireless communication corresponding to the IEEE 802.11ax (WiFi 6) standard is used for the high-speed wireless, but wireless communication corresponding to other standards, such as the WiFi 4 standard or WiFi 5 standard, may also be used. Furthermore, the display device 800 may be a device developed specifically for the camera body 1, or it may be a general-purpose smartphone or tablet device.

[0088] Furthermore, the connection between the transmitter 70 and the display device 800 may use low-power wireless communication, or it may be connected using both high-speed and low-power wireless communication, or switched between the two. In this embodiment, data with a large amount of data, such as video files of video footage described later, is transmitted using high-speed wireless communication, while lightweight data or data that can take longer to transmit is transmitted using low-power wireless communication. In this embodiment, Bluetooth is used for low-power wireless communication, but other short-range wireless communication such as NFC (Near Field Communication) may also be used.

[0089] The calibrator 850 is a device used for initial setup of the camera body 1 and for individual settings. Like the display device 800, it can connect to the transmitter 70 via high-speed wireless communication. Further details about the calibrator 850 will be described later. The display device 800 may also incorporate the functions of the calibrator 850.

[0090] The simplified display device 900 is a display device that can only be connected to the transmitter 70 via, for example, low-power wireless communication.

[0091] The simplified display device 900 cannot transmit video footage to the transmission unit 70 due to time constraints, but it can transmit the timing of the start and stop of image capture, and perform basic image confirmation such as composition checks. Furthermore, the simplified display device 900, like the display device 800, may be a device developed specifically for the camera body 1, or it may be a smartwatch or similar device.

[0092] Figure 5 is a block diagram showing the hardware configuration of camera body 1. Furthermore, the same numbers are used for the configurations and functions described using Figures 1A to 1C, etc., and detailed explanations are omitted.

[0093] In Figure 5, the camera body 1 includes an overall control CPU 101, a power switch 11, an imaging mode switch 12, a face direction detection window 13, a start switch 14, a stop switch 15, an imaging lens 16, and an LED 17.

[0094] The camera body 1 also includes an infrared LED lighting circuit 21, an infrared LED 22, an infrared focusing lens 26, and an infrared detection processing device 27, which constitute a face direction detection unit 20 (Figure 4).

[0095] Furthermore, the camera body 1 includes an imaging unit 40 (Figure 4) consisting of an imaging driver 41, a solid-state image sensor 42, and an imaging signal processing circuit 43, and a transmitting unit 70 (Figure 4) consisting of a low-power wireless unit 71 and a high-speed wireless unit 72.

[0096] In this embodiment, the camera body 1 is provided with only one shooting unit 40, but two or more shooting units 40 may be provided to capture 3D images, capture images with a wider angle of view than that obtainable with one shooting unit 40, or capture images in multiple directions.

[0097] The camera body 1 also includes various types of memory, such as a large-capacity non-volatile memory 51, a built-in non-volatile memory 102, and a primary memory 103.

[0098] Furthermore, the camera body 1 includes an audio processing unit 104, a speaker 105, a vibrator 106, an angular velocity sensor 107, an acceleration sensor 108, and various switches 110.

[0099] The overall control CPU 101 is connected to the power switch 11 and other components as shown in Figure 2C, and controls the camera body 1. The recording direction / angle determination unit 30, image cropping / development processing unit 50, and other control units 111 shown in Figure 4 are all composed of the overall control CPU 101 itself.

[0100] The infrared LED lighting circuit 21 controls the on / off state of the infrared LED 22 as described above using Figure 2E, and controls the emission of infrared light 23 from the infrared LED 22 toward the user.

[0101] The face direction detection window 13 is composed of a visible light cut filter, which blocks almost all visible light but allows sufficient transmission of infrared light 23 and its reflected light 25, which are in the infrared region.

[0102] The infrared focusing lens 26 is a lens that focuses reflected light rays 25.

[0103] The infrared detection processing device 27 (infrared detection means) has a sensor that detects reflected light rays 25 focused by an infrared focusing lens 26. This sensor forms an image of the focused reflected light rays 25, converts it into sensor data, and passes it to the overall control CPU 101.

[0104] As shown in Figure 1B, when the user is wearing the camera body 1, the face direction detection window 13 is located below the user's chin. Therefore, as shown in Figure 5, the infrared light 23 emitted from the infrared LED lighting circuit 21 passes through the face direction detection window 13 and is irradiated onto the infrared irradiation surface 24 near the user's chin. The infrared light 23 reflected by the infrared irradiation surface 24 becomes reflected light 25, which passes through the face direction detection window 13 and is focused by the infrared condensing lens 26 onto the sensor in the infrared detection processing device 27.

[0105] The various switches 110 are not shown in Figures 1A to 1C, etc., and although details are omitted, they are switches that perform functions unrelated to this embodiment.

[0106] The imaging driver 41 includes a timing generator and other components, and generates and outputs various timing signals to each part involved in imaging, thereby driving the imaging process.

[0107] The solid-state image sensor 42 outputs a signal obtained by photoelectric conversion of the subject image projected from the imaging lens 16, as explained using Figure 1A, to the imaging signal processing circuit 43.

[0108] The imaging signal processing circuit 43 performs processing such as clamping and A / D conversion on the signal from the solid-state image sensor 42 and outputs the generated imaging data to the overall control CPU 101.

[0109] The built-in non-volatile memory 102 uses flash memory or the like and stores the startup program for the overall control CPU 101 and the settings for various program modes. In this embodiment, the observation field of view (angle of view) and the effectiveness level of vibration control can be set, so these settings are also recorded.

[0110] The primary memory 103 consists of RAM and other components, and temporarily stores video data being processed, as well as the calculation results of the overall control CPU 101.

[0111] The large-capacity non-volatile memory 51 records or reads primary image data. In this embodiment, for the sake of simplicity, the description assumes that the large-capacity non-volatile memory 51 is a semiconductor memory without a removable mechanism, but it is not limited to this. For example, the large-capacity non-volatile memory 51 may be configured as a removable recording medium such as an SD card, or it may be used in combination with the built-in non-volatile memory 102.

[0112] The low-power wireless unit 71 exchanges data with the display device 800, calibrator 850, and simple display device 900 using low-power wireless communication.

[0113] The high-speed wireless unit 72 exchanges data with the display device 800, calibrator 850, and simple display device 900 using high-speed wireless communication.

[0114] The audio processing unit 104 is equipped with a microphone 19L on the right side of Figure 1A and a microphone 19R on the left side of the same figure, which pick up external sound (analog signals). It processes the picked-up analog signals and generates audio signals.

[0115] The LED 17, speaker 105, and vibrator 106 communicate or warn the user about the status of the camera body 1 by emitting light, sound, or vibration.

[0116] The angular velocity sensor 107 is a sensor that uses a gyroscope or the like, and detects the movement of the camera body 1 itself as gyro data.

[0117] The acceleration sensor 108 detects the orientation of the shooting / detection unit 10.

[0118] Furthermore, the angular velocity sensor 107 and acceleration sensor 108 are built into the imaging / detection unit 10, and a separate angular velocity sensor 807 and acceleration sensor 808 are also provided in the display device 800, which will be described later.

[0119] Figure 6 is a block diagram showing the hardware configuration of the display device 800. For the sake of simplicity, the same reference numerals are used for parts that were explained using Figure 1D, and their explanations are omitted.

[0120] In Figure 6, the display device 800 includes a display device control unit 801, button A 802, display unit 803, button B 804, in-camera 805, face sensor 806, angular velocity sensor 807, acceleration sensor 808, image capture signal processing circuit 809, and various switches 811.

[0121] The display device 800 also includes a built-in non-volatile memory 812, a primary memory 813, a large-capacity non-volatile memory 814, a speaker 815, a vibrator 816, an LED 817, an audio processing unit 820, a low-power wireless unit 871, and a high-speed wireless unit 872.

[0122] The display device control unit 801 is composed of a CPU and is connected to buttons A802 and face sensor 806, as explained using Figure 1D, and controls the display device 800.

[0123] The imaging signal processing circuit 809 performs the same functions as the imaging driver 41, solid-state image sensor 42, and imaging signal processing circuit 43 inside the camera body 1, but it is not very important for the explanation in this embodiment, so for the sake of simplicity, it is described as a single unit. The data output by the imaging signal processing circuit 809 is processed in the display device control unit 801. The details of this data processing will be described later.

[0124] The various switches 811 are not shown in Figure 1D, and details are omitted, but they are switches that perform functions unrelated to this embodiment.

[0125] The angular velocity sensor 807 is a sensor that uses a gyroscope or the like, and detects the movement of the display device 800 itself.

[0126] The accelerometer 808 detects the orientation of the display device 800 itself.

[0127] As mentioned above, the angular velocity sensor 807 and acceleration sensor 808 are built into the display device 800 and, although they have the same functions as the angular velocity sensor 107 and acceleration sensor 808 located in the camera body 1 described earlier, they are separate components.

[0128] The built-in non-volatile memory 812 uses flash memory or similar technology and stores the startup program for the display device control unit 801 and the settings for various program modes.

[0129] The primary memory 813 is composed of RAM or the like and temporarily stores video data being processed and the calculation results of the imaging signal processing circuit 809. In this embodiment, during video recording, gyro data detected by the angular velocity sensor 107 at the imaging time of each frame is associated with each frame and stored in the primary memory 813.

[0130] The high-capacity non-volatile memory 814 records or reads image data from the display device 800. In this embodiment, the high-capacity non-volatile memory 814 is configured as a removable memory, such as an SD card. Alternatively, it may be configured as a non-removable memory, such as the high-capacity non-volatile memory 51 located in the camera body 1.

[0131] The speaker 815, vibrator 816, and LED 817 communicate the status of the display device 800 to the user or provide warnings by emitting sound, vibrating, or emitting light.

[0132] The audio processing unit 820 is equipped with a left microphone 819L and a right microphone 819R that pick up external sound (analog signals), and processes the picked-up analog signals to generate audio signals.

[0133] The low-power wireless unit 871 exchanges data with the camera body 1 using low-power wireless communication.

[0134] The high-speed wireless unit 872 exchanges data with the camera body 1 using high-speed wireless communication.

[0135] The face sensor 806 (face detection means) includes an infrared LED lighting circuit 821, an infrared LED 822, an infrared focusing lens 826, and an infrared detection processing device 827.

[0136] The infrared LED lighting circuit 821 is a circuit that has the same function as the infrared LED lighting circuit 21 in Figure 5, and controls the lighting and extinguishing of the infrared LED 822 and controls the emission of infrared light 823 from the infrared LED 822 toward the user.

[0137] The infrared focusing lens 826 is a lens that focuses the reflected light rays 825 of the infrared rays 823.

[0138] The infrared detection and processing unit 827 has a sensor that detects reflected light rays focused by the infrared focusing lens 826. This sensor converts the focused reflected light rays 825 into sensor data and passes it to the display device control unit 801.

[0139] When the face sensor 806 shown in Figure 1D is pointed at the user, the infrared LED lights up as shown in Figure 6. Infrared light 823 emitted from circuit 821 is directed onto the infrared irradiation surface 824, which is the user's entire face. The infrared light 823 reflected by the infrared irradiation surface 824 becomes reflected light 825, which is then focused by the infrared focusing lens 826 onto the sensor in the infrared detection processing device 827.

[0140] The other functional unit 830, whose details are omitted here, performs functions unrelated to this embodiment, such as telephone functions and other smartphone-specific functions like sensors.

[0141] The following explains how to use the camera body 1 and the display device 800.

[0142] Figure 7A is a flowchart showing an overview of the image recording process according to this embodiment, which is performed in the camera body 1 and the display device 800.

[0143] For further explanation, Figure 7A indicates on the right side of each step which device shown in Figure 4 is performing that step. Specifically, steps S100 to S700 in Figure 7A are performed by the camera body 1, and steps S800 to S1000 in Figure 7A are performed by the display device 800.

[0144] In step S100, when the power switch 11 is turned ON and power is supplied to the camera body 1, the overall control CPU 101 starts up and reads the startup program from the built-in non-volatile memory 102. After that, the overall control CPU 101 performs preparatory operations to configure the camera body 1 before image capture. Details of the preparatory operations will be described later using Figure 7B.

[0145] In step S200, the face direction detection unit 20 detects the face direction and performs a face direction detection process to infer the observation direction. Details of the face direction detection process will be described later using Figure 7C. This process is executed at a predetermined frame rate.

[0146] In step S300, the recording direction / angle determination unit 30 performs the recording direction / range determination process. Details of the recording direction / range determination process will be described later using Figure 7D.

[0147] In step S400, the imaging unit 40 performs imaging and generates imaging data.

[0148] In step S500, the image extraction and development processing unit 50 uses the recording direction and field of view information determined in step S300 to extract the image from the imaging data generated in step S400 and performs recording range development processing on that area. Details of the recording range development processing will be described later with reference to Figure 7E.

[0149] In step S600, the primary recording unit 60 (video recording means) performs a primary recording process in which the video developed in step S500 is saved as video data in the primary memory 103. Details of the primary recording process will be described later with reference to Figure 14.

[0150] In step S700, the transmission unit 70 performs a transfer process to the display device 800, in which it wirelessly transmits the video recorded in step S600 to the display device 800 at a specified timing. Details of the transfer process to the display device 800 will be described later with reference to Figure 16.

[0151] Steps from step S800 onward are executed on the display device 800.

[0152] In step S800, the display device control unit 801 performs optical correction processing on the video transferred from the camera body 1 in step S700. Details of the optical correction processing will be described later with reference to Figure 17.

[0153] In step S900, the display device control unit 801 performs vibration damping on the image that was optically corrected in step S800. Details of the vibration damping process will be described later with reference to Figure 19.

[0154] Furthermore, the order of steps S800 and S900 can be reversed. In other words, you can perform image stabilization first and then optical correction afterward.

[0155] In step S1000, the display device control unit 801 (video recording means) performs secondary recording, recording the video, which has undergone optical correction processing and vibration damping processing in steps S800 and S900, into the large-capacity non-volatile memory 814, and then terminates this process.

[0156] Next, using Figures 7B to 7F, we will explain in detail the subroutines for each step described in Figure 7A, along with the order of processing, using other diagrams as well.

[0157] Figure 7B is a flowchart of the subroutine for the preparation operation process in step S100 of Figure 7A. This process will be explained below using the parts illustrated in Figures 2 and 5.

[0158] In step S101, it is determined whether the power switch 11 is ON or OFF. If the power remains OFF, the system waits; if it turns ON, the system proceeds to step S102.

[0159] In step S102, the mode selected by the imaging mode switch 12 is determined. As a result of the determination, if the mode selected by the imaging mode switch 12 is the video mode, the process proceeds to step S103.

[0160] In step S103, various settings for the video mode are read from the built-in non-volatile memory 102 and stored in the primary memory 103, and then the process proceeds to step S104. Here, the various settings for the video mode include the angle-of-view setting value ang (pre-set to 90° in this embodiment) and the anti-shake level specified by "strong", "medium", "off", etc.

[0161] In step S104, after starting the operation of the imaging driver 41 for the video mode, this subroutine is exited.

[0162] As a result of the determination in step S102, if the mode selected by the imaging mode switch 12 is the still image mode, the process proceeds to step S106.

[0163] In step S106, various settings for the still image mode are read from the built-in non-volatile memory 102 and stored in the primary memory 103, and then the process proceeds to step S107. Here, the various settings for the still image mode include the angle-of-view setting value ang (pre-set to 45° in this embodiment) and the anti-shake level specified by "strong", "medium", "off", etc.

[0164] In step S107, after starting the operation of the imaging driver 41 for the still image mode, this subroutine is exited.

[0165] If the result of the determination in step S102 indicates that the mode selected by the imaging mode switch 12 is the pre-set mode, the process proceeds to step S108. Here, the pre-set mode is a mode in which the imaging mode is set for the camera body 1 from an external device such as the display device 800, and is one of the three imaging modes that can be switched using the imaging mode switch 12. The pre-set mode is, in other words, a mode for custom shooting. Here, since the camera body 1 is a small wearable device, there are no operation switches or setting screens on the camera body 1 to change its detailed settings, and the settings are changed by an external device such as the display device 800. The detailed settings for the main unit 1 of the Mera device will be changed.

[0166] For example, consider a scenario where you want to capture video with both a 90° and a 110° field of view consecutively. Since the standard video mode is set to a 90° field of view, this requires first capturing video in the standard mode, then ending the video capture, switching the display device 800 to the camera body's settings screen, and switching the field of view to 110°. However, such operations on the display device 800 can be cumbersome during events.

[0167] On the other hand, if the pre-setting mode is set in advance to capture video with a 110° field of view, after capturing video with a 90° field of view, the user can instantly switch to capturing video with a 110° field of view simply by sliding the capture mode switch 12 to "Pri". In other words, the user does not need to interrupt their current activity and perform the cumbersome operation described above.

[0168] Furthermore, the settings configured in pre-setting mode may include not only the field of view, but also the image stabilization level, which can be specified as "Strong," "Medium," or "Off," as well as voice recognition settings, which are not described in this embodiment.

[0169] In step S108, the various settings for the pre-setting mode are read from the built-in non-volatile memory 102 and saved in the primary memory 103, after which the process proceeds to step S109. Here, the various settings for the pre-setting mode include the field of view setting value ang and the image stabilization level specified as "strong," "medium," or "off."

[0170] In step S109, the operation of the imaging driver 41 for the pre-setting mode is started, and then the subroutine is exited.

[0171] Here, we will explain the various video mode settings read in step S103 using Figure 13.

[0172] Figure 13 shows the menu screen for various video mode settings displayed on the display unit 803 of the display device 800 before image capture by the camera body 1. Note that the same reference numerals are used for parts identical to those in Figure 1D, and their explanation is omitted. The display unit 803 has a touch panel function, and the following explanation assumes that it functions using touch operations, including swiping.

[0173] In Figure 13, the menu screen includes a preview screen 831, a zoom lever 832, a recording start / stop button 833, a switch 834, a battery level indicator 835, a button 836, a lever 837, and an icon display unit 838.

[0174] The preview screen 831 allows you to check the image captured by the camera body 1, and to check the zoom level and field of view.

[0175] The zoom lever 832 is an operating unit that allows zoom settings to be adjusted by shifting it left or right. In this embodiment, we will describe a case where four values, 45°, 90°, 110°, and 130°, can be set as the angle of view setting value ang, but the zoom lever 832 may also be used to set values ​​other than these as the angle of view setting value ang.

[0176] The recording start / stop button 833 is a toggle switch that combines the functions of both the start switch 14 and the stop switch 15.

[0177] The switch 834 is a switch for switching the vibration prevention on and off.

[0178] The battery level indicator 835 indicates the remaining battery level of the camera body 1.

[0179] The button 836 is a button for entering other modes.

[0180] The lever 837 is a lever for setting the vibration prevention intensity. In this embodiment, only "strong" and "medium" can be set as the vibration prevention intensity, but other vibration prevention intensities, such as "weak", may also be set. Alternatively, the vibration prevention intensity may be set continuously.

[0181] The icon display unit 838 displays a plurality of thumbnail icons for preview.

[0182] FIG. 7C is a flowchart of the subroutine of the face direction detection process in step S200 of FIG. 7A. Before explaining the details of this process, the face direction detection method using infrared light projection will be described with reference to FIGS. 8A to 8K.

[0183] FIG. 8A is a diagram showing an image of the user visible from the face direction detection window 13.

[0184] The image in FIG. 8A is the same as the image captured by the visible light image sensor when there is no visible light cut filter component in the face direction detection window 13, the visible light is sufficiently transmitted, and the infrared detection processing device 27 is a visible light image sensor.

[0185] In the image of FIG. 8A, the front part of the neck 201 above the collarbone, the base of the jaw 202, the tip of the jaw 203, and the face 204 including the nose of the user are shown.

[0186] FIG. 8B is a diagram showing a case where a fluorescent lamp in the room is reflected as a background in the image of the user visible from the face direction detection window 13.

[0187] The image in Figure 8B shows multiple fluorescent lights 205 surrounding the user. As shown above, various backgrounds and other elements are reflected in the infrared detection processing unit 27 depending on the usage conditions, making it difficult for the face direction detection unit 20 and the overall control CPU 101 to isolate the face from the sensor data from the infrared detection processing unit 27. Nowadays, there are technologies that use AI to isolate such images, but these require high capabilities from the overall control CPU 101 and are not suitable for the camera body 1, which is a portable device.

[0188] In reality, the face direction detection window 13 is composed of a visible light cut filter, so it does not transmit much visible light, and therefore the image from the infrared detection processing device 27 does not look like the images in Figures 8A and 8B.

[0189] Figure 8C shows the image obtained when the user and the fluorescent lamp in the background shown in Figure 8B are imaged by the sensor of the infrared detection processing device 27 via the face direction detection window 13, with the infrared LED 22 not illuminated.

[0190] In the image in Figure 8C, the user's neck and chin appear dark. On the other hand, fluorescent lamp 205 appears somewhat brighter because it contains not only visible light but also infrared components.

[0191] Figure 8D shows an image obtained when the user and the fluorescent lamp in the background shown in Figure 8B are imaged by the sensor of the infrared detection processing device 27 through the face direction detection window 13, with the infrared LED 22 turned on.

[0192] In the image in Figure 8D, the user's neck and chin are illuminated. On the other hand, unlike in Figure 8C, fluorescence... The brightness around light 205 remains unchanged.

[0193] Figure 8E shows the difference image calculated from the images in Figures 8C and 8D. The user's face can be seen.

[0194] In this way, the overall control CPU 101 (image acquisition means) calculates the difference between the images formed by the infrared detection processing device 27's sensor when the infrared LED 22 is lit and when it is not, thereby obtaining a difference image (hereinafter also referred to as a face image) in which the user's face is extracted.

[0195] In this embodiment, the face direction detection unit 20 employs a method of acquiring face images by extracting infrared reflection intensity as a two-dimensional image using the infrared detection processing unit 27. The sensor of the infrared detection processing unit 27 employs a structure similar to that of a general image sensor and acquires face images one frame at a time. The vertical synchronization signal (hereinafter referred to as the V signal) for frame synchronization is generated by the infrared detection processing unit 27 and output to the overall control CPU 101.

[0196] Figure 9 is a timing chart showing the timing of the infrared LED 22 turning on and off.

[0197] Figure 9(a) shows the timing at which the V signal is generated by the infrared detection processing unit 27. When the V signal becomes Hi, the timing of frame synchronization and the on / off of the infrared LED 22 is determined.

[0198] In Figure 9(a), t1 represents the period for the first facial image acquisition, and t2 represents the period for the second facial image acquisition. Figures 9(a), (b), (c), and (d) are presented so that their horizontal time axes are identical.

[0199] Figure 9(b) shows the H position of the image signal output from the sensor of the infrared detection processing unit 27 on the vertical axis. The infrared detection processing unit 27 controls the movement of its sensor so that the H position of the image signal is synchronized with the V signal, as shown in Figure 9(b). As mentioned above, the sensor of the infrared detection processing unit 27 employs a structure similar to that of a general image sensor, and its movement is well known, so the detailed control is omitted.

[0200] Figure 9(c) shows the switching timing between Hi and Low of the IR-ON signal output from the overall control CPU 101 to the infrared LED lighting circuit 21. The switching between Hi and Low of the IR-ON signal is controlled by the overall control CPU 101 in synchronization with the V signal, as shown in Figure 9(c). Specifically, the overall control CPU 101 outputs a Low IR-ON signal to the infrared LED lighting circuit 21 during period t1, and outputs a Hi IR-ON signal to the infrared LED lighting circuit 21 during period t2.

[0201] Here, while the IR-ON signal is Hi, the infrared LED lighting circuit 21 lights up the infrared LED 22, and infrared light 23 is projected onto the user. On the other hand, while the IR-ON signal is Low, the infrared LED lighting circuit 21 turns off the infrared LED 22.

[0202] Figure 9(d) shows the imaging data output from the infrared detection and processing unit 27 sensor to the overall control CPU 101. The vertical axis represents the signal intensity and indicates the amount of reflected light 25 received. In other words, during period t1, the infrared LED 22 is off, so there is no reflected light 25 from the user's face, and imaging data like that shown in Figure 8C is obtained. On the other hand, during period t2, the infrared LED 22 is on, so there is reflected light 25 from the user's face, and imaging data like that shown in Figure 8D is obtained. Therefore, as shown in Figure 9(d), the signal intensity during period t2 is greater than the signal intensity during period t1 due to the amount of reflected light 25 from the user's face. The strength increases only.

[0203] Figure 9(e) shows the difference between the imaging data during periods t1 and t2 in Figure 9(d), resulting in imaging data in which only the component of reflected light 25 from the user's face is extracted, as shown in Figure 8E.

[0204] Figure 7C shows the face direction detection process in step S200, including the operations described using Figures 8C to 8E and Figure 9 above.

[0205] First, in step S201, when the V signal output from the infrared detection processing device 27 becomes timing V1, which is the start of period t1, the process proceeds to step S202.

[0206] Next, in step S202, the IR-ON signal is set to Low and output to the infrared LED lighting circuit 21. As a result, the infrared LED 22 is turned off.

[0207] In step S203, the imaging data for one frame output from the infrared detection processing device 27 during the period t1 is read out and temporarily stored as Frame1 in the primary memory 103.

[0208] In step S204, when the V signal output from the infrared detection processing unit 27 reaches timing V2, which marks the start of period t2, the process proceeds to step S203.

[0209] In step S205, the IR-ON signal is set to Hi and output to the infrared LED lighting circuit 21. As a result, the infrared LED 22 lights up.

[0210] In step S206, the imaging data for one frame output from the infrared detection processing unit 27 during the period t2 is read out and temporarily stored as Frame2 in the primary memory 103.

[0211] In step S207, the IR-ON signal is set to Low and output to the infrared LED lighting circuit 21. This turns off the infrared LED 22.

[0212] In step S208, Frame1 and Frame2 are read from the primary memory 103, and the difference obtained by subtracting Frame1 from Frame2 is used to calculate the light intensity Fn of the 25 components of the user's reflected light rays in Figure 9(e) (this is generally known as the blacking process).

[0213] In step S209, the neck position (center of neck rotation) is extracted from the light intensity Fn.

[0214] First, the overall control CPU 101 (division means) divides the face image into multiple distance areas, which will be explained using Figure 8F, based on the light intensity Fn.

[0215] Figure 8F shows the case where the density of the difference image in Figure 8E is adjusted to match the light intensity of the reflected light rays 25 of the infrared 23 projected onto the user's face and neck, in order to see the distribution of light intensity for each part of the user's face and neck.

[0216] Figure 8F(a) is a diagram showing the distribution of light intensity of reflected light rays 25 in the facial image of Figure 8E, divided into regions and shown in gray to simplify the explanation. For explanatory purposes, the Xf axis is taken in the direction from the center of the user's neck to the tip of the chin.

[0217] Figure 8F(a) shows the light intensity on the Xf axis in Figure 8F(a) on the horizontal axis, and the Xf axis on the vertical axis. The horizontal axis indicates light intensity, with increasing intensity towards the right.

[0218] In Figure 8F(a), the facial image is divided into six regions (distance areas) 211-216 according to light intensity.

[0219] Region 211 is the region with the strongest light intensity and is shown in white as a gray area.

[0220] Region 212 is a region where the light intensity is slightly lower than that of region 211, and is shown as a gray area, specifically in a fairly light gray color.

[0221] Region 213 is a region where the light intensity is even lower than that of region 212, and is shown as a gray area, represented by a light gray color.

[0222] Region 214 is a region where the light intensity is even lower than that of region 213, and is shown as an intermediate gray color, representing a gray range.

[0223] Region 215 is a region where the light intensity is even lower than that of region 214, and is shown as a gray area, represented by a slightly darker gray color.

[0224] Region 216 is the region with the weakest light intensity and is the darkest shade of gray. Above region 216, there is no light intensity and it is black.

[0225] This light intensity will be explained in detail below using Figure 10.

[0226] Figure 10 illustrates the vertical movement of the user's face, showing the user's position as observed from the left side.

[0227] Figure 10(a) shows the user facing forward. The imaging and detection unit 10 is located in front of the user's collarbone. In addition, infrared light 23 from the infrared LED 22 is irradiated onto the lower part of the user's head from the face direction detection window 13 located above the imaging and detection unit 10. If we let Dn be the distance from the face direction detection window 13 to the base of the neck 200 above the user's collarbone, Db be the distance from the face direction detection window 13 to the base of the chin 202, and Dc be the distance from the face direction detection window 13 to the tip of the chin 203, then it can be seen that the distances increase in the order of Dn, Db, and Dc. Since light intensity is inversely proportional to the square of the distance, the light intensity when the reflected light 25 from the infrared irradiation surface 24 is imaged by the sensor of the infrared detection processing device 27 decreases in the order of the base of the neck 200, the base of the chin 202, and the tip of the chin 203. Furthermore, it can be seen that the light intensity of the face 204, including the nose, which is located at a distance greater than Dc from the face direction detection window 13, becomes even dimmer. In other words, in a case like Figure 10(a), it can be seen that an image with the light intensity distribution shown in Figure 8F is acquired.

[0228] Furthermore, the configuration of the face direction detection unit 20 is not limited to the configuration shown in this embodiment, as long as the direction of the user's face can be detected. For example, an infrared pattern may be irradiated from the infrared LED 22 (infrared pattern irradiation means), and the infrared pattern reflected from the irradiated object may be detected by the sensor (infrared pattern detection means) of the infrared detection processing device 27. In this case, the sensor of the infrared detection processing device 27 is preferably a structural light sensor. Alternatively, the sensor of the infrared detection processing device 27 may be a sensor that performs phase comparison between infrared rays 23 and reflected light rays 25 (infrared phase comparison means), for example, a ToF sensor.

[0229] Next, using Figure 8G, we will explain the extraction of the neck position in step S209 of Figure 7C.

[0230] Figure 8G(a) is a diagram in which the symbols indicating the various parts of the user's body in Figure 10(a), as well as the symbols of the double circle and black circle indicating the neck and chin positions, are superimposed on Figure 8F.

[0231] The white area 211 corresponds to the base of the neck 200 (Figure 10(a)), the fairly light gray area 212 corresponds to the front of the neck 201 (Figure 10(a)), and the light gray area 213 corresponds to the base of the chin 202 (Figure 10(a)). The medium gray area 214 corresponds to the tip of the chin 203 (Figure 10(a)), and the slightly darker gray area 215 corresponds to the lower part of the face 204 (Figure 10(a)), specifically the lips and the surrounding lower face. Furthermore, the darker gray area 216 corresponds to the upper part of the face 204 (Figure 10(a)), specifically the nose and the surrounding upper face.

[0232] Furthermore, as shown in Figure 10(a), the distance between Db and Dc is small compared to the distance from the face direction detection window 13 to other parts of the user, so the difference in reflected light intensity between the light gray region 213 and the medium gray region 214 is also small.

[0233] On the other hand, as shown in Figure 10(a), the distance Dn is the shortest distance among the distances from the face direction detection window 13 to each part of the user, so the white area 211 corresponding to the base of the neck 200 is the area with the strongest reflectivity.

[0234] Therefore, the overall control CPU 101 (setting means) sets the position 206, indicated by a double circle in Figure 8G(a), which is the center of the left and right sides of region 211 and closest to the imaging / detection unit 10, as the neck rotation center position (hereinafter referred to as neck base position 206). This process is what is done in step S209 of Figure 7C.

[0235] Next, using Figure 8G, we will explain the extraction of the chin position in step S210 of Figure 7C.

[0236] As shown in Figure 8G(a), the intermediate gray region 214, which is brighter than the region 215 corresponding to the lower part of the face including the lips within the face 204, is the region including the chin. As can be seen in Figure 8G(b), the light intensity drops sharply in the region 215 adjacent to region 214, and the change in distance from the face direction detection window 13 becomes large. The overall control CPU 101 determines that the region 214 in front of the region 215 where there is a sharp drop in light intensity is the chin region. Furthermore, the overall control CPU 101 calculates (extracts) the chin position 207 at the left-right center of region 214 and the position furthest from the neck position 206 (the position shown by the black circle in Figure 8G(a)).

[0237] For example, Figures 8H and 8I show the changes when the face is turned to the right.

[0238] Figure 8H shows the difference image calculated in the same way as in Figure 8E when the user's face is turned to the right. Figure 8I is a diagram in which the double circle and black circle symbols indicating the neck position 206 and chin position 207r, which are the center positions of neck movement, are superimposed on Figure 8H.

[0239] Because the user is facing right, region 214 moves to region 214r, shown in Figure 8I, which is to the left when viewed from the side of the shooting / detection unit 10. Region 215, which corresponds to the lower part of the face including the lips within face 204, also moves to region 215r, which is to the left when viewed from the side of the shooting / detection unit 10.

[0240] Therefore, the overall control CPU 101 identifies the region 214r in front of 215r, where there is a sharp drop in light intensity, as the chin region. Furthermore, the overall control CPU 101 identifies the position at the center of 214r and furthest from the neck position 206 (the position shown by the black circle in Figure 8I) as the chin position. Calculate (extract) using location 207r.

[0241] Subsequently, the overall control CPU 101 determines the movement angle θr, which indicates how far the chin position 207r in Figure 8I has moved, centered on the neck position 206, from the chin position 207 in Figure 8G(a) to the right. As shown in Figure 8I, the movement angle θr is the angle in the left-right direction of the user's face.

[0242] In step S210, the infrared detection processing device 27 of the face direction detection unit 20 (3D detection sensor) detects the position of the chin and the left-right angle of the user's face using the method described above.

[0243] Next, we will explain the detection of the face in the upward direction.

[0244] Figure 10(b) shows the user with their face turned horizontally, and Figure 10(c) shows the user with their face turned 33° above the horizontal.

[0245] In Figure 10(b), the distance from the face direction detection window 13 to the chin tip 203 is denoted as Ffh, and in Figure 10(c), the distance from the face direction detection window 13 to the chin tip 203u is denoted as Ffu.

[0246] As shown in Figure 10(c), the chin tip 203u moves upward along with the face, so it can be seen that the distance between Ffu and Ffh is longer.

[0247] Figure 8J shows the image of the user as seen through the face direction detection window 13 when the user's face is turned 33° above the horizontal. As shown in Figure 10(c), the user is looking upwards, so the face 204, including the lips and nose, is not visible from the face direction detection window 13 located below the user's chin, and only the tip of the chin 203 is visible. Figure 8K shows the distribution of the light intensity of the reflected light rays 25 when infrared rays 23 are irradiated onto the user at this time. Figure 8K is a differential image calculated in the same way as in Figure 8E, with double circles and black circles indicating the neck position 206 and the chin position 207u superimposed.

[0248] The six regions 211u to 216u in Figure 8K, corresponding to light intensity, are regions with the same light intensity as the region shown in Figure 8F, but with "u" added to them. In Figure 8F, the light intensity of the user's chin 203 was in the intermediate gray region 214, but in Figure 8K, it has shifted towards the gray side and is in the slightly darker gray region 215u. Thus, as shown in Figure 10(c), the infrared detection and processing device 27 can detect that, as a result of Ffu being at a longer distance than Ffh, the light intensity of the reflected light 25 from the user's chin 203 is weakened inversely proportional to the square of the distance.

[0249] Next, we will explain the detection of the face in the downward direction.

[0250] Figure 10(d) shows the user with their face tilted 22° downward from the horizontal.

[0251] In Figure 10(d), Ffd is defined as the distance from the face direction detection window 13 to the chin tip 203d.

[0252] As shown in Figure 10(d), since the chin tip 203d moves downward along with the face, the distance of Ffd becomes shorter than that of Ffh, and the light intensity of the reflected light rays 25 from the chin tip 203 becomes stronger.

[0253] Returning to Figure 7C, in step S211, the overall control CPU 101 (distance calculation means) calculates the distance from the chin position to the face direction detection window 13 based on the light intensity of the chin position detected by the infrared detection processing device 27 of the face direction detection unit 20 (3D detection sensor). Based on this, the upper part of the face The downward angle is also calculated.

[0254] In step S212, the angles of the face in the left-right direction (first detection direction) and the vertical direction perpendicular to it (second detection direction), acquired in steps S210 and S211 respectively, are stored in the primary memory 103 as the user's observation direction vi, which consists of three dimensions (i is an arbitrary sign). For example, if the user was observing the center of the front, the observation direction vo would be the vector information [0°,0°], since the left-right direction θh is 0° and the vertical direction θv is 0°. Also, if the user was observing 45° to the right, the observation direction vr would be the vector information [45°,0°].

[0255] In step S211, the vertical angle of the face was calculated by detecting the distance from the face direction detection window 13, but this method is not limited to this method. For example, the angle change may be calculated by comparing the level of variation in the light intensity of the chin tip 203. In other words, the angle change of the chin may be calculated based on the gradient change of the gradient CDu of the reflected light intensity from the chin base 202 to the chin tip 203 in Figure 8K(c), relative to the gradient CDh of the reflected light intensity from the chin base 202 to the chin tip 203 in Figure 8G(a).

[0256] Figure 7D is a flowchart of the subroutine for determining the recording direction and recording range in step S300 of Figure 7A. Before explaining the details of this process, we will first describe the ultra-wide-angle video for which the recording direction and recording range are determined in this embodiment, using Figure 11A.

[0257] In this embodiment, the camera body 1 achieves the acquisition of an image in the observation direction by having the shooting unit 40 capture an ultra-wide-angle image around the shooting / detection unit 10 using an ultra-wide-angle imaging lens 16, and then cropping a portion of that image.

[0258] Figure 11A shows the target field of view 125 in the ultra-wide-angle image captured by the shooting unit 40 when the user is facing forward.

[0259] As shown in Figure 11A, the image-capable pixel area 121 of the solid-state image sensor 42 is a rectangular area. The effective projection area 122 (predetermined area) is the area where a circular hemispherical image projected onto the solid-state image sensor 42 by the imaging lens 16 is displayed. The imaging lens 16 is adjusted so that the center of the pixel area 121 and the center of the effective projection area 122 coincide.

[0260] The outermost edge of the circular effective projection area 122 indicates the position with an FOV (Field of View) angle of 180°. When the user is looking at the horizontal and vertical center, the target field of view 125, which is the area to be imaged and recorded, has an angle of 90° from the center of the effective projection area 122, which is half that angle. In this embodiment, the imaging lens 16 can also introduce light rays from outside the effective projection area 122, and can project light rays up to a maximum FOV angle of approximately 192° onto the solid-state image sensor 42 using a fisheye projection. However, beyond the effective projection area 122, the optical performance deteriorates significantly, with extreme drops in resolution, light intensity, and distortion. Therefore, in this embodiment, the recording area will be explained using an example where the image in the observation direction is extracted only from the image projected onto the pixel area 121 of the hemispherical image displayed on the effective projection area 122 (hereinafter simply referred to as the ultra-wide-angle image).

[0261] In this embodiment, the vertical size of the effective projection area 122 is larger than the shorter side size of the pixel area 121, so the images at the upper and lower edges of the effective projection area 122 are outside the pixel area 121, but this is not limited to this. For example, the configuration of the imaging lens 16 may be changed to design the effective projection area 122 so that the entire area of ​​the effective projection area 122 fits within the area of ​​the pixel area 121.

[0262] The invalid pixel region 123 is the pixel region of the pixel region 121 that was not included in the effective projection region 122.

[0263] The target field of view 125 is the area that extracts the image in the user's viewing direction from the ultra-wide-angle image, and is defined by a preset left, right, up, and down field of view (here, 45°, FOV angle 90°) centered on the viewing direction. In the example in Figure 11A, the user is facing forward, so the center of the target field of view 125 is the viewing direction vo, which is the center of the effective projection area 122.

[0264] The ultra-wide-angle image shown in Figure 11A includes subject A131, which is a child; subject B132, which is a staircase that the child (subject A) is about to climb; and subject C133, which is a toy train shaped like a locomotive.

[0265] Next, Figure 7D shows the recording direction and range determination process in step S300, which is performed to obtain an image of the observation direction from the ultra-wide-angle image described using Figure 11A above. The following explanation of this process will be given using Figures 12A to 12G, which are specific examples of the target field of view 125.

[0266] In step S301, the previously set field of view setting value ang is obtained by reading it from the primary memory 103.

[0267] In this embodiment, all field of view angles that allow the image in the observation direction to be extracted from the ultra-wide-angle image by the image extraction / development processing unit 50—45°, 90°, 110°, and 130°—are stored as field of view setting values ​​ang in the built-in non-volatile memory 102. In addition, in any of steps S103, S106, or S108, one of the field of view setting values ​​ang stored in the built-in non-volatile memory 102 is set and stored in the primary memory 103.

[0268] Furthermore, in step S301, the observation direction vi determined in step S212 is set as the recording direction, and the image of the target field of view 125, which is extracted from the ultra-wide-angle image with the acquired field of view setting value ang centered on this direction, is saved to the primary memory 103.

[0269] For example, if the field of view setting value ang is 90° and the observation direction vo (vector information [0°,0°]) is detected by the face direction detection process (Figure 7C), the target field of view 125 (Figure 11A) is set to a range of 45° to the left and right and 45° up and down, centered on the center O of the effective projection unit 122. In other words, the overall control CPU 101 (relative position setting means) sets the angle of the face direction detected by the face direction detection unit 20 to the observation direction vi, which is vector information indicating the relative position to the ultra-wide-angle image.

[0270] In this case, when the observation direction is vo, the effect of optical distortion due to the imaging lens 16 can be almost ignored, so the shape of the set target field of view 125 becomes the shape of the target field of view 125o (Figure 12A) after distortion conversion in step S303, which will be described later. Hereafter, the target field of view 125 after distortion conversion in the case of the observation direction vi will be called the target field of view 125i.

[0271] Next, in step S302, the pre-set vibration isolation level is obtained by reading it from the primary memory 103.

[0272] In this embodiment, as described above, the vibration isolation level is set in one of steps S103, S106, or S108 and stored in the primary memory 103.

[0273] Furthermore, in step S302, the amount of spare pixels Pis for vibration damping is set based on the vibration damping level obtained above.

[0274] In the vibration stabilization process, the amount of shake of the shooting / detection unit 10 is tracked, and an image is acquired that tracks the image in the opposite direction to the shake direction. For this reason, in this embodiment, the area around the target field of view 125i is necessary for vibration stabilization. A reserve area will be provided.

[0275] In this embodiment, a table is stored in the built-in non-volatile memory 102 that holds the value of the vibration-damping reserve pixel count Pis associated with each vibration-damping level. For example, if the vibration-damping level is "medium", a reserve pixel area of ​​100 pixels, which is the vibration-damping reserve pixel count Pis read from the table, is set as the reserve area.

[0276] Figure 12E shows an example where a reserve area is added around the target field of view 125o shown in Figure 12A. Here, we will explain the case where the vibration stabilization level is "medium," that is, the vibration stabilization reserve pixel amount Pis is 100 pixels.

[0277] As shown in Figure 12E, the dotted lines in the upper, lower, left, and right directions of the target field of view 125o, each providing a margin (reserve area) of 100 pixels, which is the amount of spare image-stabilizing pixels Pis, represent the spare image-stabilizing pixel frame 126o.

[0278] Figures 12A and 12E illustrate the case where the observation direction vi coincides with the center O of the effective projection area 122 (the optical axis center of the imaging lens 16) for the sake of simplicity. However, as will be explained in the following steps, if the observation direction vi is in the peripheral area of ​​the effective projection area 122, it is affected by optical distortion and therefore conversion is necessary.

[0279] In step S303, the shape of the target field of view 125 set in step S301 is corrected (distortion converted) considering the observation direction vi and the optical characteristics of the imaging lens 16 to generate the target field of view 125i. Similarly, the number of spare pixels Pi for vibration damping set in step S302 is also corrected considering the observation direction vi and the optical characteristics of the imaging lens 16.

[0280] For example, suppose the field of view setting value ang is 90° and the user is observing 45° to the right of the center o. In this case, the observation direction vi determined in step S212 is the observation direction vr (vector information [45°, 0°]), and the target field of view 125 is the range of 45° to the left and right and 45° up and down, centered on the observation direction vr. However, considering the optical characteristics of the imaging lens 16, the target field of view 125 is corrected to the target field of view 125r shown in Figure 12B.

[0281] As shown in Figure 12B, the target field of view 125r widens towards the periphery of the effective projection area 122, and the position of the observation direction vr is also slightly inward from the center of the target field of view 125r. This is because, in this embodiment, the imaging lens 16 has an optical design similar to that of a stereoscopic fisheye. Note that if the imaging lens 16 is designed as an equidistant fisheye, equisolid angle fisheye, or orthographic fisheye, this relationship will change, and corrections will be made to the target field of view 125 according to its optical characteristics.

[0282] Figure 12F shows an example in which a reserve area with the same vibration isolation level ("medium") as the reserve area in Figure 12E is added around the target field of view 125r shown in Figure 12B.

[0283] In the vibration-damping reserve pixel frame 126o (Figure 12E), a margin of 100 pixels, which is the vibration-damping reserve pixel number Pis, is set for each of the top, bottom, left, and right sides of the target field of view 125o. In contrast, in the vibration-damping reserve pixel frame 126r (Figure 12F), the vibration-damping reserve pixel number Pis is corrected and increases as you move towards the periphery of the effective projection area 122.

[0284] Thus, similar to the shape of the target field of view 125r, the shape of the reserve area necessary for vibration damping provided around it also increases in correction amount towards the periphery of the effective projection area 122, as shown in the vibration damping reserve pixel frame 126r in Figure 12F. This is because, in this embodiment, the imaging lens 16 has an optical design similar to that of a stereoscopic projection fisheye. Note that the imaging lens 16 is an equidistant projection The relationship changes depending on the design, such as with fisheye lenses, equisolid angle projection fisheye lenses, or orthographic projection fisheye lenses, so corrections are made to the 126r spare pixel frame for image stabilization to match the optical characteristics of the respective lens.

[0285] The process performed in step S303, which sequentially switches the shape of the target field of view 125 and its reserve area considering the optical characteristics of the imaging lens 16, is a complex process. Therefore, in this embodiment, the process in step S303 is performed using a table stored in the built-in non-volatile memory 102 that holds the shape of the target field of view 125i and its reserve area for each observation direction vi. Depending on the optical design of the imaging lens 16 mentioned above, the calculation formula may be stored in the overall control CPU 101, and the optical distortion value may be calculated using that formula.

[0286] In step S304, the position and size of the video recording frame are calculated.

[0287] As described above, in step S303, a reserve area necessary for vibration isolation was targeted and placed around the field of view 125i, and this was calculated as the vibration isolation reserve pixel frame 126i. However, depending on the position in the observation direction vi, its shape becomes quite unusual, for example, as in the vibration isolation reserve pixel frame 126r.

[0288] The overall control CPU 101 can extract the image by performing development processing only on the area with this special shape. However, it is not common to use a non-rectangular image when recording it as image data in step S600 or transferring it to the display device 800 in step S700. Therefore, in step S304, the position and size of the rectangular image recording frame 127i that encompasses the entire vibration-damping spare pixel frame 126i are calculated.

[0289] Figure 12F shows the video recording frame 127r, indicated by a dashed line, which was calculated in step S304 relative to the vibration-damping spare pixel frame 126r.

[0290] In step S305, the position and size of the video recording frame 127i calculated in step S304 are recorded in the primary memory 103.

[0291] In this embodiment, the upper left coordinates Xi,Yi of the video recording frame 127i in the ultra-wide-angle image are recorded as the position of the video recording frame 127i, and the width WXi and height WYi of the video recording frame 127i from coordinates Xi,Yi are recorded as the size of the video recording frame 127i. For example, for the video recording frame 127r shown in Figure 12F, the shown coordinates Xr,Yr, width WXr, and height WYr are recorded in step S305. Note that the coordinates Xi,Yi are XY coordinates with a predetermined reference point, specifically the optical center of the imaging lens 16, as the origin.

[0292] Once the vibration-damping spare pixel frame 126i and the video recording frame 127i have been determined in this way, the subroutine of step S300 shown in Figure 7D is exited.

[0293] Up to this point, in order to simplify the explanation of the complex optical distortion transformation, we have used observation directions vi that include horizontal 0°, i.e., observation direction vo (vector information [0°,0°]) and observation direction vr (vector information [45°,0°]) as examples of observation direction vi. However, in reality, the user's observation direction vi will be in various directions. Therefore, the recording range development process performed in such cases will be explained below.

[0294] For example, when the field of view setting value ang is 90° and the observation direction vl[-42°,-40°], the target field of view 125l will be as shown in Figure 12C.

[0295] Furthermore, even with the same observation direction vl (vector information [-42°,-40°]) as the target field of view 125l, if the field of view setting value ang is 45°, the target field of view 128l will be slightly smaller than the target field of view 125l, as shown in Figure 12D. Regarding the target field of view 128l, As shown in Figure 12G, a vibration-damping spare pixel frame 129l and a video recording frame 130l are set up.

[0296] Step S400 is a basic imaging operation and uses a general sequence for the imaging unit 40, so details are left to other literature and will be omitted here. In this embodiment, the imaging signal processing circuit 43 in the imaging unit 40 also performs processing to correct the signal output from the solid-state image sensor 42, which is in a specific output format (examples of standards: MIPI, SLVS), into imaging data using a general sensor readout method.

[0297] Furthermore, if the mode selected by the imaging mode switch 12 is video mode, the shooting unit 40 starts recording when the start switch 14 is pressed. Recording then ends when the stop switch 15 is pressed. On the other hand, if the mode selected by the imaging mode switch 12 is still image mode, the shooting unit 40 takes a still image each time the start switch 14 is pressed.

[0298] Figure 7E is a flowchart of the subroutine for the recording range development process in step S500 of Figure 7A.

[0299] In step S501, the raw data of the entire area of ​​the imaging data (ultra-wide-angle video) generated by the imaging unit 40 in step S400 is acquired and input to the video acquisition unit called the head unit (not shown) of the overall control CPU 101.

[0300] Next, in step S502, based on the coordinates Xi, Yi, width WXi, and height WYi recorded in the primary memory 103 in step S305, the portion of the video recording frame 127i is extracted from the ultra-wide-angle video acquired in step S501. After this extraction, the cropping and development process (Figure 7F), consisting of steps S503 to S508, is started only on the pixels within the vibration-damping reserve pixel frame 126i. This significantly reduces the amount of computation compared to performing development on the entire area of ​​the ultra-wide-angle video read in step S501, thereby reducing computation time and power consumption.

[0301] Furthermore, as shown in Figure 7F, if the mode selected by the imaging mode switch 12 is video mode, the processes in steps S200 and S300 and the process in step S400 are executed in parallel at the same or different frame rates. In other words, each time the Raw data for the entire area of ​​one frame generated by the imaging unit 40 is acquired, cropping and development processing is performed based on the coordinates Xi, Yi, width WXi, and height WYi recorded in the primary memory 103 at that time.

[0302] When the crop development process for pixels within the vibration-damping spare pixel frame 126i is started, first, in step S503, color interpolation is performed to complete the color pixel information arranged in the Bayer array.

[0303] After that, the white balance is adjusted in step S504, and then the color conversion is performed in step S505.

[0304] In step S506, gamma correction is performed to correct the gradation according to a pre-set gamma correction value.

[0305] In step S507, edge enhancement is performed according to the image size.

[0306] In step S508, the data is converted into a data format that can be temporarily stored by compressing or other processing, recorded in primary memory 103, and then the subroutine is exited. Details about the data format will be explained later.

[0307] Furthermore, the order and whether or not the cropping and development processes performed in steps S503 to S508 are performed may be adjusted according to the camera system and do not limit the present invention.

[0308] Furthermore, if video mode is selected, the process from steps S200 to S500 will be repeated until recording is finished.

[0309] This process significantly reduces the amount of computation required compared to processing the entire area read in step S501. As a result, an inexpensive and low-power microcontroller can be used as the overall control CPU 101, and heat generation in the overall control CPU 101 is suppressed, while the battery life of the battery 94 is also improved.

[0310] Furthermore, in this embodiment, in order to reduce the control load on the overall control CPU 101, the optical correction processing (step S800 in Figure 7A) and vibration damping processing (step S900 in Figure 7A) of the image are not performed by the camera body 1, but are transferred to the display device 800 and then performed by the display device control unit 801. Therefore, if only the image data partially extracted from the projected ultra-wide-angle image is sent to the display device 800, the optical correction processing and vibration damping processing cannot be performed. In other words, the extracted image data alone does not contain the position information used to substitute into formulas during optical correction processing or to refer to correction tables during vibration damping processing, so these processes cannot be correctly executed in the display device 800. For this reason, in this embodiment, not only the extracted image data but also correction data including information on the extraction position from the ultra-wide-angle image is sent from the camera body 1 to the display device 800.

[0311] If the extracted image is a still image, the still image data and correction data are transmitted separately to the display device 800, but since there is a one-to-one correspondence between the still image data and the correction data, the display device 800 can correctly perform optical correction processing and image stabilization processing. On the other hand, if the extracted image is a video, transmitting the video data and correction data separately to the display device 800 makes it difficult to determine which correction data corresponds to which frame of the video. In particular, if the clock rate of the overall control CPU 101 in the camera body 1 and the clock rate of the display device control unit 801 in the display device 800 are slightly different, synchronization between the overall control CPU 101 and the display device control unit 801 becomes impossible after several minutes of video capture. As a result, the display device control unit 801 may correct the frame that should be processed with correction data different from the corresponding correction data, leading to problems such as this.

[0312] Therefore, in this embodiment, when transmitting video data extracted from the camera body 1 to the display device 800, correction data is appropriately added to the video data. The method for doing so will be described below.

[0313] Figure 14 is a flowchart of the subroutine for the primary recording process in step S600 of Figure 7A. This process will be explained below with reference to Figure 15. Figure 14 shows the process when the mode selected by the imaging mode switch 12 is video mode. If the selected mode is still image mode, this process starts from step S601 and ends when step S606 is completed.

[0314] In step S601a, the overall control CPU 101 reads an image of one frame from the video footage developed in the recording range development process (Figure 7E) that has not been processed in steps S601 to S606. The overall control CPU 101 (metadata generation means) also generates correction data, which is metadata for the read frame.

[0315] In step S601, the overall control CPU 101 attaches information about the image extraction position of the frame read in step S600 to the correction data. The information attached here is the coordinates Xi,Yi of the video recording frame 127i acquired in step S305. Alternatively, the information attached here may be vector information indicating the observation direction Vi.

[0316] In step S602, the overall control CPU 101 (optical correction value acquisition means) acquires an optical correction value. The optical correction value is the optical distortion value set in step S303. Alternatively, it may be a correction value corresponding to the lens optical characteristics, such as peripheral light intensity correction value or diffraction correction value.

[0317] In step S603, the overall control CPU 101 attaches the optical correction values ​​used for distortion conversion in step S602 to the correction data.

[0318] In step S604, the overall control CPU 101 determines whether or not the camera is in vibration damping mode. Specifically, if the pre-set vibration damping mode is "medium" or "strong," it determines that the camera is in vibration damping mode and proceeds to step S605. On the other hand, if the pre-set vibration damping mode is "off," it determines that the camera is not in vibration damping mode and proceeds to step S606. The reason for skipping step S605 when the vibration damping mode is "off" is that this reduces the amount of calculation data used by the overall control CPU 101 and the amount of data transmitted wirelessly, which in turn reduces the power consumption and heat generation of the camera body 1. Although the reduction of data used for vibration damping processing has been explained here, it is also possible to reduce data such as peripheral light intensity correction values ​​and whether or not analytical correction is present, which are included in the optical correction values ​​acquired in step S602.

[0319] In this embodiment, the vibration isolation mode is pre-set by user operation via the display device 800, but it may also be set as the initial setting of the camera body 1. Furthermore, if the camera system is configured to switch between vibration isolation and non-vibration isolation after data is transmitted to the display device 800, step S604 can be omitted, and the system proceeds directly from step S603 to step S605.

[0320] In step S605, the overall control CPU 101 (movement detection means) attaches the vibration isolation mode acquired in step S302 and the gyro data during video capture, which is linked to the frame read in step S600 and stored in the primary memory 813, to the correction data.

[0321] In step S606, the video file 1000 (Figure 15) is updated with data encoded from the image data of the frame read in step S600 and the correction data to which various data have been attached in steps S601 to S605. If the first frame of the video footage was read in step S601a, the video file 1000 is generated in step S606.

[0322] In step S607, it is determined whether the reading of all frames of the video developed in the recording range development process (Figure 7E) has finished. If not, the process returns to step S601a. ​​If it has finished, the subroutine is exited. The generated video file 1000 is saved in the built-in non-volatile memory 102. In addition to being saved in the primary memory 813 and the built-in non-volatile memory 102 as described above, it may also be saved in the large-capacity non-volatile memory 51. Alternatively, the generated video file 1000 may be immediately transferred to the display device 800 (step S700 in Figure 7A), and after being transferred to the display device 800, it may be saved in its primary memory 813.

[0323] In this embodiment, encoding refers to combining video data and correction data into a single file. However, the video data may be compressed, or the combined video data and correction data may be compressed.

[0324] Figure 15 shows the data structure of video file 1000.

[0325] The video file 1000 consists of a header 1001 and a frame 1002. Frame 1002 is composed of a frame dataset, which is a set of images for each frame that makes up the video and their corresponding frame metadata. In other words, frame 1002 contains as many frame datasets as there are frames in the video.

[0326] In this embodiment, the frame metadata is information encoded with correction data, which includes the cropping position (position information within the image), optical correction value, and gyro data as needed, but is not limited to this. For example, the amount of information in the frame metadata may be changed by adding other information to the frame metadata or deleting information in the frame metadata depending on the imaging mode selected by the imaging mode switch 12.

[0327] Header 1001 records the offset value or starting address to the frame dataset for each frame. Alternatively, metadata such as the time and size corresponding to the video file 1000 may be stored.

[0328] Thus, in the primary recording process (Figure 14), a video file 1000 containing each frame of the video developed in the recording range development process (Figure 7E) and its metadata is transferred to the display device 800. Therefore, even if the clock rate of the overall control CPU 101 of the camera body 1 and the clock rate of the display device control unit 801 of the display device 800 are slightly different, the display device control unit 801 can reliably perform the correction process of the video developed by the camera body 1.

[0329] In this embodiment, the optical correction value was included in the frame metadata, but the optical correction value may be applied to the entire video.

[0330] Figure 16 is a flowchart of the subroutine for the transfer process to the display device 800 in step S700 of Figure 7A. Figure 16 shows the process when the mode selected by the imaging mode switch 12 is video mode. If the selected mode is still image mode, this process starts from step S702.

[0331] In step S701, it is determined whether the recording of video footage by the shooting unit 40 (step S400) has finished or is still in progress. If video footage is being recorded (video capture in progress), the recording range development process for each frame (step S500) and the updating of the video file 1000 in the primary recording process (step S600) (step S606) are performed sequentially. Wireless transfer has a high power load, so performing it in parallel with recording would require a large battery capacity for the battery 94 and separate measures to prevent overheating. Also, from the perspective of computing power, performing wireless transfer in parallel with recording increases the computing load, so it is necessary to prepare a high-spec overall control CPU 101, which also increases the cost. In this embodiment, taking these factors into consideration, the system waits for the video footage recording to finish (YES in step S701) before proceeding to step S702 to establish a connection with the display device 800. However, if the camera system of this embodiment has sufficient power supplied from the battery 94 and no additional heat dissipation measures are required, the camera body 1 may be connected to the display device 800 in advance, such as when it is started up or before recording begins.

[0332] In step S702, a connection to the display device 800 is established via the high-speed wireless unit 72 in order to transfer the video file 1000, which has a large amount of data, to the display device 800. The low-power wireless unit 71 is used for transferring low-resolution video (or video) to the display device 800 for checking the field of view, and for sending and receiving various setting values ​​between the display device 800 and the display device 800, but transmission takes time. Because it requires a certain amount of processing time, it is not used for transferring 1000 video files.

[0333] In step S703, the video file 1000 is transferred to the display device 800 via the high-speed wireless unit 72. Once the transfer is complete, the process proceeds to step S704, where the connection with the display device 800 is closed, and then the subroutine is exited.

[0334] Up to this point, we have described the case of transferring a single video file containing images of all frames of a single video. However, for long video videos lasting several minutes, it is also acceptable to use multiple video files divided by time units. If the video file has the data structure shown in Figure 15, even if a single video is transferred to the display device 800 as multiple video files, the display device 800 can correct the video without any timing discrepancies with the correction data.

[0335] Figure 17 is a flowchart of the subroutine for the optical correction process in step S800 of Figure 7A. This process will be explained below with reference to Figure 18. As mentioned above, this process is executed by the display device control unit 801 of the display device 800.

[0336] In step S801, the display device control unit 801 (video file receiving means) first receives the video file 1000 from the camera body 1 that was transferred in the transfer process to the display device 800 (step S700). Subsequently, the display device control unit 801 (first extraction means) obtains the optical correction value extracted from the received video file 1000.

[0337] Next, in step S802, the display device control unit 801 (second extraction means) acquires video (an image of one frame obtained by video capture) from the video file 1000.

[0338] In step S803, the display device control unit 801 (frame image correction means) performs optical correction of the image acquired in step S802 using the optical correction value acquired in step S801, and saves the corrected image to the primary memory 813. When performing optical correction, if cropping is performed from the image acquired in step S802, the image is cropped and processed in a range narrower than the development range (target field of view 125i) determined in step S303 (cropped development area).

[0339] Figure 18 is a diagram illustrating the case where distortion correction is performed in step S803 of Figure 17.

[0340] Figure 18(a) shows the position of the subject 1401 as seen by the user with the naked eye during imaging, and Figure 18(b) shows the image of the subject 1401 projected onto the solid-state image sensor 42.

[0341] Figure 18(c) shows the development region 1402 in the image of Figure 18(b). Here, the development region 1402 is the cropped development region explained earlier.

[0342] Figure 18(d) shows the cropped developed area, from which the image of the developed area 1402 has been extracted, and Figure 18(e) shows the image obtained by correcting the distortion of the cropped developed area in Figure 18(d). Since cropping is performed during the distortion correction of the cropped developed image, the field of view of the image shown in Figure 18(e) is even smaller than that of the cropped developed area shown in Figure 18(d).

[0343] Figure 19 is a flowchart of the vibration isolation subroutine in step S900 of Figure 7A. This process will be explained below with reference to Figure 25. As mentioned above, this process is executed by the display device control unit 801 of the display device 800.

[0344] In step S901, the gyro data for the current frame and the previous frame are obtained from the frame metadata of video file 1000, and the data for the previous frame was calculated in step S902, which will be described later. Blur amount V n-1 Det Obtain the following information. Then, from this information, estimate the amount of deviation V n Pr e This is calculated. In this embodiment, the current frame is the frame currently being processed, and the previous frame is the frame immediately preceding the current frame.

[0345] Step S902 provides detailed information on the amount of blur V from the video. n Det This is how the amount of blur is determined. Blur detection is performed by calculating how much the feature points of the current frame's image have moved from the previous frame.

[0346] Known methods can be adopted for feature point extraction. For example, a luminance information image can be generated by extracting only the luminance information of the frame image, and an image shifted by 1 to several pixels can be subtracted from the original image, and pixels with an absolute value greater than or equal to a threshold can be extracted as feature points. Alternatively, an image obtained by applying a high-pass filter to the luminance information image can be subtracted from the original luminance information image, and the extracted edges can be extracted as feature points.

[0347] The movement amount is calculated by calculating the difference multiple times while shifting the luminance information images of the current frame and the previous frame by 1 to several pixels each, and calculating the position where the difference at the pixels of the feature points decreases.

[0348] Since multiple feature points are required as described later, it is preferable to divide the images of the current frame and the previous frame into a plurality of blocks and perform feature point extraction. The block division depends on the number of pixels and the aspect ratio of the image, but generally 12 blocks of 4×3 to 96×64 blocks are preferable. If the number of blocks is small, correction for trapezoid due to the sway of the imaging unit 40 of the camera body 1 or rotation in the optical axis direction cannot be accurately performed. If the number of blocks is too large, the size of one block becomes small and the feature points become close to each other, resulting in errors. For these reasons, an optimal number of blocks is appropriately selected according to the number of pixels, the ease of finding feature points, the shooting angle of the subject, etc.

[0349] To calculate the movement amount, it is necessary to shift the luminance information images of the current frame and the previous frame by 1 to several pixels each and perform multiple difference calculations, resulting in a large amount of calculation. Therefore, the actual movement amount is the blur amount V n Pre To calculate how many pixels it is shifted from V, the amount of calculation can be significantly reduced by performing difference calculations only in the vicinity thereof.

[0350] Next, in step S903, after performing shake correction using the detailed blur amount V obtained in step S902, this subroutine is exited. n Det After performing shake correction using the detailed blur amount V obtained in step S902, this subroutine is exited.

[0351] Furthermore, conventional methods for vibration isolation include Euclidean transforms that allow rotation and translation, affine transforms that allow these actions, and projective transforms that allow trapezoidal correction.

[0352] While movement and rotation along the X and Y axes can be corrected using Euclidean transformations, the blur that occurs when actually capturing images with the camera body 1's shooting unit 40 also includes camera shake in the front-to-back direction and in the pan-tilt direction. Therefore, in this embodiment, vibration correction is performed using affine transformation, which can also correct for magnification and skew. In affine transformation, when the coordinates (x,y) of a reference feature point move to coordinates (x',y'), it is expressed by the following equation 100.

[0353]

number

[0354] Figure 18(f) shows the image after applying the image stabilization correction in step S903 to the distortion-corrected image shown in Figure 18(e). Because cropping is performed during image stabilization correction, the field of view of the image shown in Figure 18(f) is smaller than that of the image shown in Figure 18(e).

[0355] By performing this type of image stabilization, it is possible to obtain high-quality images with corrected blur.

[0356] The above describes a series of operations performed by the camera body 1 and the display device 800 included in the camera system of this embodiment.

[0357] After the user turns on the power switch 11 and selects the video mode with the imaging mode switch 12, and simply observes the front without turning their face up, down, left, or right, the face direction detection unit 20 first detects the observation direction vo (vector information [0°,0°]) (Figure 12A). Then, the recording direction / angle determination unit 30 extracts the image of the target field of view 125o shown in Figure 12A (Figure 11B) from the ultra-wide-angle image projected onto the solid-state image sensor 42.

[0358] Subsequently, without the user operating the camera body 1, if, for example, the user starts observing the child (subject A131) in Figure 11A, the face direction detection unit 20 first detects the observation direction vl (vector information [-42°, -40°]) (Figure 11C). Then, the recording direction / angle determination unit 30 extracts the image with the target field of view 125l (Figure 11C) from the ultra-wide-angle image captured by the shooting unit 40.

[0359] In this way, optical correction and vibration damping processing are performed on the display device 800 in steps S800 and S900 for the images cropped into various shapes according to the observation direction. As a result, even if the overall control CPU 101 of the camera body 1 has low specifications, even when cropping an image with significant distortion, such as the target field of view 125l (Figure 11C), it is possible to obtain an image with distortion and shaking corrected, centered on the child (subject A131), as shown in Figure 11D. In other words, the user can obtain an image captured in their observation direction without touching the camera body 1, other than turning on the power switch 11 and selecting a mode with the imaging mode switch 12.

[0360] Now, let's explain the pre-configuration mode. As mentioned above, the camera body 1 is a small wearable device, so it does not have any operation switches or setting screens for changing its detailed settings. Therefore, the detailed settings of the camera body 1 are changed using an external device such as the display device 800 (in this embodiment, the setting screen of the display device 800 (Figure 13)).

[0361] For example, consider a scenario where you want to capture video with both a 90° and a 45° field of view consecutively. In normal video mode, the field of view is set to 90°, so to perform this type of capture, you would first need to capture video in normal video mode, then stop the video capture, switch the display device 800 to the camera body 1 settings screen, and switch the field of view to 45°. However, performing such operations on the display device 800 during continuous capture is cumbersome, and you might miss capturing the footage you want.

[0362] On the other hand, if you pre-set the pre-setting mode to the mode for capturing video with a 45° field of view, After capturing video at a 90° angle, simply sliding the imaging mode switch 12 to "Pri" instantly switches to zoomed-in video capture at a 45° angle. In other words, the user does not need to interrupt the current imaging process and perform the aforementioned cumbersome operation.

[0363] The settings configured in pre-setting mode may include not only changes to the field of view, but also the image stabilization level, which can be specified as "strong," "medium," or "off," as well as changes to the voice recognition settings, which are not described in this embodiment.

[0364] For example, in the aforementioned imaging situation, if the user continues to observe the child (subject A131) and switches from video mode to pre-setting mode using the imaging mode switch 12, the field of view setting value ang changes from 90° to 45°. In this case, the recording direction / field of view determination unit 30 extracts an image of the target field of view 128l, shown by the dotted frame in Figure 11E, from the ultra-wide-angle image captured by the shooting unit 40.

[0365] Even in pre-setting mode, optical correction processing and image stabilization processing are performed on the display device 800 in steps S800 and S900. This allows the camera body 1's overall control CPU 101 to have low specifications, while still images with corrected distortion and shaking can be obtained, zoomed in on the child (subject A131), as shown in Figure 11F. The example shown was changing the field of view setting value ang from 90° to 45° in video mode, but the same applies to still image mode. The same also applies when the video field of view setting value ang is 90° and the still image field of view setting value ang is 45°.

[0366] In this way, the user can obtain a zoomed-in image of their own observation direction simply by switching modes using the imaging mode switch 12 on the camera body 1.

[0367] In this embodiment, the case in which the face direction detection unit 20 and the shooting unit 40 are integrally configured in the camera body 1 has been described. However, this is not limited to the case in which the face direction detection unit 20 is mounted on the user's body other than their head, and the shooting unit 40 is mounted on the user's body. For example, the shooting / detection unit 10 of this embodiment can also be installed on the shoulder or abdomen. However, in the case of the shoulder, if the shooting unit 40 is installed on the right shoulder, the subject on the left side may be obstructed by the head, so it is preferable to install multiple shooting means, including on the left shoulder, to compensate. Also, in the case of the abdomen, a spatial parallax occurs between the shooting unit 40 and the head, so it is desirable to be able to perform a correction calculation of the observation direction to correct for that parallax, as shown in Embodiment 3.

[0368] (Example 2) In Example 2, a method for calibrating individual differences and adjustment differences among users who attach the camera body 1 will be explained in detail using Figures 20 to 23.

[0369] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 2, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0370] Users who wear the camera body 1 have individual differences and adjustment differences, such as their physique, the tilt and angle of the neck area where the camera body 1 is attached, the condition of their clothing, such as the collar, when wearing it, and the remaining adjustment of the band parts 82L and 82R. Therefore, the optical axis center of the imaging lens 16 of the camera body 1 and the field of view center when the user is facing forward (hereinafter referred to as the user's natural state) do not usually coincide. For the user, it is desirable to set the center of the recording area (target field of view 125) in which the image is captured, rather than setting the optical axis center of the imaging lens 16 of the camera body 1 directly as the center of the recording area (target field of view 125).

[0371] Furthermore, there are individual differences not only in the user's natural field of view center, but also in the field of view center when the user turns their head in any direction, including up, down, left, right, and diagonally, as well as in the range of motion of the neck. Therefore, there are individual differences in the relationship between the face direction (observation direction) detected by the face direction detection unit 20 and the center position of the target field of view 125 (hereinafter referred to as the field of view center position) set according to that observation direction. Consequently, calibration work is required to associate the face direction with the field of view center position.

[0372] Ideally, the calibration operation should be performed as part of the preparation process (step S100) shown in Figure 7A. While it is usually assumed that the calibration operation will be performed when the camera body 1 is first started up, it may also be performed when a certain amount of time has elapsed since the last calibration, or when the camera body 1 has shifted position relative to the user since the last calibration. The calibration operation may also be performed when the face direction detection unit 20 can no longer detect the user's face. Furthermore, if the system detects that the user has attached or detached the camera body 1, the calibration operation may be performed when the user reattaches it. In this way, it is desirable that the calibration operation be performed as appropriate at the timing deemed necessary for the proper use of the camera body 1.

[0373] Figure 20 shows the details of the calibrator 850 used in the calibration process according to Example 2. In this example, the case in which the calibrator 850 also functions as a display device 800 will be described.

[0374] The calibrator 850 includes the components of the display device 800 shown in Figure 7D: button A802, display unit 803, in-camera 805, face sensor 806, and angular velocity sensor 807, as well as a positioning index 851 and a calibration button 854. Note that button B804, which was used in Example 1, is not used in this embodiment and can be replaced by the calibration button 854 as described later, and is therefore not shown here.

[0375] Figure 20(a) shows the case where the positioning index 851 is a specific pattern displayed on the display unit 803, while Figure 20(b) shows the case where the external appearance of the calibrator 850 is used for the positioning index 851. In this case, the positioning index center 852, which will be described later, is calculated from the information of the external shape of the calibrator 850.

[0376] Furthermore, the positioning index 851 is not limited to the examples shown in Figures 20(a) and (b). For example, it may be a separate component from the calibrator 850. Anything is acceptable as long as it is easy to measure in size and has a shape that easily points towards the user's face. For example, it could be the lens cap of the imaging lens 16 or the charging unit of the camera body 1. In any case, the basic concept of the calibration operation is the same, so the explanation below will mainly use the calibrator 850 shown in Figure 20(a) as an example.

[0377] As also mentioned in Example 1, the calibrator 850 may also function as a display device 800. Furthermore, like the display device 800, the calibrator 850 may be a dedicated device, or it may be a general-purpose smartphone or tablet device.

[0378] The positioning index 851 is an index displayed on the display unit 803 of the calibrator 850, and is a figure that allows for the calculation of the width L851a, the height L851b, and the center 852 of the positioning index. In the calibration process described later, the user directs their face towards the vicinity of the center of the positioning index 851, so it is desirable that the positioning index 851 has a shape that is easy to grasp in the center of the field of view. In Figure 20(a), it is shown as a circle with a black circle in the center of a cross, but it is not limited to this shape. Other shapes such as a square are also possible. It could be a triangular or star-shaped figure, or even an illustration of a character.

[0379] The positioning index 851 is captured by the imaging unit 40 of the camera body 1. Based on the captured image, the display device control unit 801 (position calculation means and distance calculation means) calculates the distance between the imaging / detection unit 10 and the calibrator 850, and the position coordinates of the positioning index 851 that are captured within the image range. In this embodiment, such calculations are performed by the display device 800 which is integrated with the calibrator 850, but if the calibrator 850 or the positioning index 851 is separate from the display device 800, such calculations are performed by the overall control CPU 101 on the camera body 1 side.

[0380] The angular velocity sensor 807 can measure the movement of the calibrator 850. Based on the measurement values ​​from the angular velocity sensor 807, the display device control unit 801 calculates movement information indicating the position and orientation of the calibrator 850, which will be described later.

[0381] The calibration button 854 is a button that the user presses when they face the center of the positioning indicator 851 during the calibration process described later. In Figure 20(a), the calibration button 854 is a touch button displayed on the touch panel display unit 803, but buttons A802 and B804 may also function as the calibration button 854.

[0382] Next, the calibration process, which is performed when extracting an image from the ultra-wide-angle image captured by the shooting unit 40 according to the user's face direction and performing image processing on that image, as described in Example 1, will be explained in detail using the flowchart in Figure 21.

[0383] Figure 21 is a flowchart of the calibration process according to this embodiment, which is performed in the camera body 1 (first calibration means) and the calibrator 850.

[0384] For illustrative purposes, in Figure 21, the steps in which the camera body 1 and calibrator 850 receive user input are placed in a frame where the user is the primary operator. Also in Figure 21, the steps executed by the display device control unit 801 of the calibrator 850 in response to the user input are placed in a frame where the calibrator 850 is the primary operator. Similarly, in Figure 21, the steps executed by the overall control CPU 101 of the camera body 1 in response to the user input are placed in a frame where the camera body 1 is the primary operator.

[0385] Specifically, in steps S3104 and S3108 of Figure 21, the camera body 1 is the primary operator, while in steps S3101, S3105, and S3106, the user is the primary operator. Furthermore, in steps S3102, S3103, S3107, and S3110, the calibrator 850 is the primary operator.

[0386] When this process begins, in step S3101, if the calibrator 850 is not powered on, the user operates button A802 to power on the calibrator 850. Similarly, if the camera body 1 is not powered on, the user switches the power switch 11 to ON to power on the camera body 1. After that, the user establishes a connection between the calibrator 850 and the camera body 1. Once this connection is established, the display unit 801 and the overall control CPU 101 each enter calibration mode.

[0387] In step S3101, the user attaches the camera body 1, adjusts the length of the bands 82L and 82R and the angle of the camera body 1, and positions the camera body 1 in a suitable location so that the shooting / detection unit 10 can capture images.

[0388] In step S3102, the display device control unit 801 (first display means) displays the positioning index The number 851 is displayed on the display unit 803.

[0389] Next, in step S3103, the display control unit 801 instructs the user on the instruction display 855 to indicate the positions (each designated position) where the calibrator 850 should be held. In this embodiment, five locations are indicated as designated positions in order: the front, upper right, lower right, upper left, and lower left. However, the designated positions are not limited to these, as calibration is possible.

[0390] In step S3104, the overall control CPU 101 activates the imaging unit 40 to enable image acquisition, and also activates the face direction detection unit 20 to enable detection of the user's face direction.

[0391] In step S3105, the user holds the calibrator 850 over the designated position indicated in step S3103.

[0392] Next, in step S3106, the user turns their face toward the positioning indicator 851 while maintaining the position of the calibrator 850 in the designated position, and places the center of the user's field of view on the positioning indicator 851.

[0393] In step S3107, the display control unit 801 (second display means) notifies the user via the instruction display 855 that calibration of the specified position will begin, and also displays the calibration button 854. The process in step S3107 is executed when the display control unit 801 determines that the user has viewed the positioning index center 852 of the positioning index 851 in the front of their field of view.

[0394] In step S3107a, when the user presses the calibration button 854, the display device control unit 801 sends a calibration instruction to the camera body 1.

[0395] In step S3108, the overall control CPU 101 (acquisition / detection means) acquires an ultra-wide-angle image in which the positioning index 851 is captured by the imaging unit 40 in response to a calibration instruction from the calibrator 850, and simultaneously detects the face direction with the face direction detection unit 20. Subsequently, the overall control CPU 101 (generation means) calculates the position coordinate information of the center 852 of the positioning index in the ultra-wide-angle image acquired here, and generates information showing the relationship between the calculated position coordinate information and the face direction detected here.

[0396] The details of the processes in steps S3103 to S3108 will be explained below using Figures 22A to 22F.

[0397] Figures 22A to 22F illustrate the calibration operation in the direction directly in front of the user. This ensures that the user's natural field of view center position matches the center position of the target field of view 125 in the image captured by the camera unit 40 of the camera body 1.

[0398] Figure 22A shows the screen displayed on the display unit 803 of the calibrator 850 in step S3103 of Figure 21 during the calibration operation in the direction facing the user.

[0399] As shown in Figure 22A, the display unit 803 of the calibrator 850 displays a positioning indicator 851 and an instruction display 855 that shows where the user should place the positioning indicator 851.

[0400] Instruction display 855 contains textual instructions to position the positioning indicator 851 at the center of the field of view when the face is facing forward. Instructions are not limited to those shown; other methods of instruction, such as illustrations, photographs, or videos, are also acceptable.

[0401] Alternatively, a typical tutorial approach could be used, where instruction indicator 855 is displayed first, followed by positioning indicator 851.

[0402] Figure 22B is a perspective view showing a user holding the calibrator forward in accordance with the instructions shown in Figure 22A.

[0403] The user holds the calibrator 850 forward according to the instructions shown on the instruction display 855 in Figure 22A, positioning the calibrator 850 so that the positioning indicator 851 is at the center of the field of view when facing the face (steps S3105, S3106). Then, the user presses the calibration button 854 (Figure 22A). The determination in step S3107 is made in response to the pressing of the calibration button 854. The specific procedure for this determination method will be described later.

[0404] After that, the user confirms that the instruction display 855 shown in Figure 22A has changed to a notification that says "Starting forward calibration," and then presses the calibration button 854 (step S3107a).

[0405] In response to the pressing of the calibration button 854, the imaging unit 40 acquires the captured image in step S3108.

[0406] Figure 22C is a schematic diagram showing the entire ultra-wide-angle image captured by the imaging lens 16 in the state shown in Figure 22B, and Figure 22D is a schematic diagram showing the image after correcting the aberrations of the ultra-wide-angle image shown in Figure 22C.

[0407] Meanwhile, in step S3108, the face direction detection unit 20 acquires the face direction in response to the user pressing the calibration button 854 in the state shown in Figure 22B.

[0408] Figure 22E is a schematic diagram showing the face direction image recorded by the face direction detection unit 20 in step S3108 of Figure 21 during the calibration operation for the user's front direction.

[0409] As described above using Figures 8G to 8K in Example 1, the face direction detection unit 20 calculates the left-right and up-down angles of the face using the distance and angle between the chin position 207, 207r, 207u, etc. and the neck position 206. However, the distance and angle values ​​between the chin position 207, 207r, 207u, etc. and the neck position 206 are not constant, as are individual differences and adjustment differences, such as those represented by the user's physique, as mentioned above, similar to the image center. Therefore, in this embodiment, the relationship between the chin position and the neck position 206 at the time the calibration button 854 is pressed is defined as the value when the user's front is the center of their field of view. This makes it possible to use this information to accurately calculate the user's face direction regardless of individual differences or adjustment differences.

[0410] Returning to Figure 21, in step S3109, the overall control CPU 101 determines whether the preparation for frontal calibration is complete. That is, it determines whether the information necessary for calculating the chin position 207, the neck position 206, and the positioning index center 852 has been acquired.

[0411] If the necessary information has not been obtained at this time, it is determined that the preparation for calibration in the forward direction is not complete (NO in step S3109), and the operation from step S3102 is repeated to obtain the missing information. If the information has not been obtained, it is naturally not necessary to perform all the operations from step S3102 onwards; only the operations necessary to obtain the missing information again may be performed again.

[0412] Here, the determination in step S3107 is made using the face sensor 806 or the in-camera 805 mounted on the calibrator 850. The specific procedure for this determination method will be explained below using the in-camera 805 as an example to show the calibration operation for the user's front direction. Note that the case where the face sensor 806 is used is omitted because although there is a difference in whether the information is two-dimensional or three-dimensional, the basic concept is the same. However, when the face sensor 806 is used for the determination in step S3107, the face direction detection unit 20 of the camera body 1 does not perform face detection by emitting infrared 23 to the user while infrared 823 is being projected onto the user from the face sensor 806. This is to prevent the infrared 23 and 823 from interfering with each other.

[0413] First, when the user presses the calibration button 854 in Figure 22A in step S3106, the display device control unit 801 takes an image with the in-camera 805 (face detection means) and acquires an in-camera image 858 (Figure 22F) showing the user. Furthermore, the display device control unit 801 detects the user's face 204, including the front of the neck 201, the tip of the chin 203, and the nose, as well as the position information of the shooting / detection unit 10 (shooting unit 40), from the acquired in-camera image 858.

[0414] Using the positional information detected in the in-camera image 858, the display device control unit 801 (determination means) determines whether the user is looking at the positioning index center 852 of the positioning index 851 in the front of their field of view.

[0415] Furthermore, if this determination determines that the user is looking in a different direction, the display device control unit 801 will display information on the instruction display 855 indicating that correct information could not be obtained. This allows the system to instruct the user to repeat the calibration operation.

[0416] Furthermore, the display control unit 801 may determine, using the in-camera image 858, that the shooting / detection unit 10 is tilted beyond a certain point, or that the face direction detection window 13 is blocked or dirty, or that other conditions prevent proper calibration. In such cases, the display control unit 801 may also display information on the instruction display 855 indicating that correct information cannot be acquired.

[0417] Furthermore, it is possible to obtain the information necessary for parallax correction, which will be described later in Example 5, using the in-camera image 858 acquired in step S3107 and the ultra-wide-angle image acquired in step S3108.

[0418] Specifically, before the positioning index 851 is captured by the imaging unit 40 in step S3108, information on the size of the positioning index 851 (width L851a and height L851b) is transmitted in advance from the display device 800 to the camera body 1. This allows the overall control CPU 101 to calculate the distance between the imaging / detection unit 10 and the positioning index 851 using the information on the size of the positioning index 851 and the image of the positioning index 851 captured in the ultra-wide-angle image acquired in step S3108. The positioning index 851 is located on the calibrator 850, which has the same housing as the in-camera 805, and in Figure 22B, the calibrator 850 is facing the user almost directly, so it can be seen that the in-camera 805 and the imaging / detection unit 10 are also at the same distance.

[0419] Similarly, before the in-camera image shown in Figure 22F is captured by the in-camera 805 in step S3107a, information about the size of the shooting / detection unit 10 is transmitted in advance from the camera body 1 to the display device 800. This allows the display device control unit 801 (vertical distance calculation means) to perform the shooting / detection. Using the size information of the detection unit 10 and the image of the shooting / detection unit 10 captured in the in-camera image 858 in Figure 22F, the vertical distance 5070 between the optical axis center of the imaging lens 16 and the user's viewpoint position can be estimated. In addition, the display device control unit 801 can also estimate the distance 2071 between the imaging lens 16 and the user's chin 203. The distance 2071 may also be the distance between the face direction detection window 13 and the chin 203.

[0420] Here, for the face direction detection unit 20 to calculate the user's neck position 206 and chin position, the user's face must be at a certain distance or more from the face direction detection window 13, according to the design intent of the face direction detection unit 20. Therefore, this estimation result can be used as one of the criteria for determining whether the face direction detection unit 20 can correctly detect the face direction.

[0421] Returning to Figure 21, if, in step S3109, the overall control CPU 101 determines that it has acquired the necessary information and that the preparation for calibration in the forward direction is complete, it proceeds to step S3110.

[0422] In step S3110, the display unit control unit 801 (first calibration means) calculates the information necessary to offset the cutting center position in order to absorb individual differences and adjustment differences, and offsets the cutting center position based on that information.

[0423] The specific details of the calculation in step S3110 are as follows:

[0424] If the user is in an ideal state according to the design specifications and the camera body 1 is ideally attached, the center 856 of the ultra-wide-angle image acquired in step S3108 shown in Figure 22C and the position of the positioning index center 852 in that ultra-wide-angle image should approximately coincide. However, in reality, due to individual differences and adjustment differences such as the user's physique as described above, the positions of the center 856 and the positioning index center 852 in the ultra-wide-angle image usually do not coincide.

[0425] For the user, the cropping center position should preferably be the position of the positioning index center 852 in the ultra-wide-angle image, which corresponds to the user's posture and movement, rather than the image center indicated by the camera body 1, i.e., the center 856.

[0426] Therefore, the amount of displacement 857 between the positioning index center 852 and the center 856 in the ultra-wide-angle image is measured, and the cropping center position is offset to a value based on the positioning index center 852 rather than the center 856 in the camera body 1. The face direction detected by the face direction detection unit 20 is also offset in the same manner.

[0427] As for the specific offset method, as shown in Figure 22C, the displacement amount 857 is measured for the ultra-wide-angle image, and this is divided into a horizontal displacement amount 857a and a vertical displacement amount 857b. After performing an appropriate transformation process according to the projection method for the entire field of view, the offset amount can be determined.

[0428] Alternatively, as shown in Figure 22D, the offset amount may be determined after performing an appropriate transformation process on the ultra-wide-angle image according to the projection method. That is, the amount of displacement 857a (not shown in Figure 22D) between the center 856a and the positioning index center 852a in the transformed ultra-wide-angle image may be measured, and the offset amount may be determined by dividing the displacement amount 857a into a left-right displacement amount 857c and a up-down displacement amount 857d.

[0429] The choice of offset method, as shown in Figure 22C or Figure 22D, can be arbitrarily determined by considering the processing load and purpose of the camera system.

[0430] By performing the forward-facing calibration operation described above, individual differences and adjustment differences will be compensated for. Furthermore, it becomes possible to appropriately associate the face direction of each user when wearing the device with the center of the field of view in that face direction within the ultra-wide-angle image and the face direction of the face direction detection unit 20.

[0431] Up to this point, we have explained the calibration procedure for the front direction, out of the five directions: front, upper right, lower right, upper left, and lower left. However, the same calibration procedure must also be performed for the four directions: upper right, lower right, upper left, and lower left.

[0432] Therefore, in Figure 21, once the processing in step S3110 is completed, the process proceeds to step S3111.

[0433] In step S3111, if it is determined that there is one of the five directions (front, upper right, lower right, upper left, lower left) in which the calibration operation has not yet been performed, the direction in which the calibration operation is performed is changed to that one direction, and the process returns to step S3103. This repeats the calibration operation for the remaining directions, excluding the front direction which has already been completed.

[0434] Although not shown in Figure 21, if it is determined in step S3111 that there are no directions where calibration has not been performed, this process is terminated.

[0435] Figures 23A to 23E illustrate the calibration operation in the direction of the user's right hand upwards (upper right direction in ultra-wide-angle images). Figures 23A to 23E correspond to Figures 22A to 22E, respectively, and the basic operation is the same, so common explanations are omitted.

[0436] Here, as shown in Figure 23A, the instruction display 855 contains textual instructions to position the positioning indicator 851 at the center of the field of view when the face is turned to the upper right.

[0437] Figure 23B is a perspective view showing the user holding the calibrator 850 to the upper right in accordance with the instructions shown on the instruction indicator 855 in Figure 23A.

[0438] Figure 23C is a schematic diagram showing the entire ultra-wide-angle image captured by the imaging lens 16 in the state shown in Figure 23B.

[0439] As shown in Figure 23C, the specific offset method involves first measuring the displacement 857 between the center 856 and the positioning index center 852 in the ultra-wide-angle image. Then, the measured displacement 857 is divided into a diametrical displacement 857e and an angular displacement 857f, and the offset amount is determined after performing an appropriate transformation process according to the projection method for the entire field of view.

[0440] Alternatively, as shown in Figure 23D, the offset amount may be determined after performing an appropriate transformation process on the ultra-wide-angle image according to the projection method. That is, the amount of displacement 857a (not shown in Figure 23D) between the center 856a and the positioning index center 852a in the transformed ultra-wide-angle image may be measured, and the offset amount may be determined by dividing the displacement amount 857a into a displacement amount in the diametrical direction 857g and a displacement amount in the angular direction 857h.

[0441] In the method of determining the offset amount explained using Figures 22A to 22E, a method was used to divide the displacement into vertical and horizontal directions. In contrast, in the method of determining the offset amount explained using Figures 23A to 23D, a method was used to divide the displacement into diameter and angular directions. However, this difference in method is merely for the sake of explanation, and either method may be used.

[0442] Furthermore, as shown in Figure 23E, the face direction detection unit 20 can obtain the neck position 206 and chin position 207ru, which are necessary for calculating the face direction when the user is facing upwards to the right. Therefore, regardless of individual differences or adjustment differences among users, it is possible to correctly measure the direction of the user's face when they are looking in the direction of the positioning index center 852 (in this case, the upper right direction).

[0443] As described above, in the calibration process shown in Figure 21, calibration operations are performed not only in the front direction but also in the upper right, lower right, upper left, and lower left directions. This allows the face direction detection unit 20 to correctly measure which direction the user is facing when the user turns their head in any of the up, down, left, or right directions, enabling the camera body 1 to be used appropriately regardless of individual differences or adjustment differences.

[0444] In the above description, for simplicity, we explicitly explained how to perform repeated calibration operations in five directions: front, upper right, lower right, upper left, and lower left.

[0445] However, the calibration operation is not limited to this method. For example, the user may continuously move the calibrator 850 in shapes such as a Z-shape, spiral, or polygon according to the instruction display 855, while simultaneously keeping the positioning index 851 displayed on the calibrator 850 in the center of the field of view. In this method, the display device control unit 801 sends calibration instructions to the camera body 1 multiple times while the calibrator 850 is moving in this manner. Each time the overall control CPU 101 receives a calibration instruction, it acquires the face direction detected by the face direction detection unit 20 and the position coordinate information of the center of the positioning index 852 in the ultra-wide-angle image captured by the shooting unit 40, and stores this as history information. Subsequently, the overall control CPU 101 combines the information extracted from the acquired history information to calculate the relationship between the center position of the image crop and the face direction of the user. Furthermore, using the information from the in-camera 805 and face sensor 806 acquired by the calibrator 850 during its movement using this method, the information extracted from the history data may be limited to only the information indicating that the user is looking at the positioning indicator 851. This prevents information such as when the user is looking away from the history data, thereby improving the accuracy of relationship calculations.

[0446] Furthermore, the display device control unit 801 may also transmit the measurement value from the angular velocity sensor 807 to the camera body 1 when a calibration command is issued. In this case, the overall control CPU 101 acquires movement information from the transmitted measurement value from the angular velocity sensor 807, indicating how the user moves the calibrator 850 and the position and orientation of the calibrator 850, and also stores this as history information. This makes it possible to perform the calibration operation simply and accurately using the movement information based on the measurement value from the angular velocity sensor 807, the face direction detected by the face direction detection unit 20, and the position coordinate information of the positioning index center 852 in the ultra-wide-angle image captured by the shooting unit 40.

[0447] However, in this case, the movement information based on the measurement value from the angular velocity sensor 807 and the movement information based on the position coordinate information from the positioning index 851 must match. Therefore, when using the measurement value from the angular velocity sensor 807, it is necessary to synchronize the communication between the camera body 1 and the calibrator 850.

[0448] In Example 2, a calibration method was described that relates the user's face direction to the center position of the target field of view 125 in the ultra-wide-angle image, even with individual differences and adjustment differences. However, the present invention is not limited to the various forms exemplified in Example 2, and various modifications and changes are possible within the scope of its gist.

[0449] (Example 3) In Example 3, a method for preventing motion sickness caused by secondarily recorded video will be explained using Figures 24-26.

[0450] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 3, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0451] Thanks to advancements in video technology, it's now easy to enjoy CG that's indistinguishable from live-action footage and impressive 3D images.

[0452] On the other hand, when such 3D images are dynamic or shaky, such as those found in VR, viewers are more likely to experience motion sickness. Motion sickness can cause symptoms similar to motion sickness, and there is growing interest in safety measures to address this.

[0453] If the recording range development process (step S500) is performed by simply cutting out and developing the image in the direction the user's face is facing, then if the user's face moves quickly during imaging by the shooting unit 40 (step S400), the video scene will also switch at a rapid speed.

[0454] While the user themselves may not experience motion sickness due to rapid facial movements during imaging by the camera unit 40, if the resulting video, which was secondarily recorded in step S1000, includes such scenes, viewers watching the video may experience motion sickness.

[0455] The video captures the direction the face is pointing. Japanese Patent Publication No. 2007-74033, Japanese Patent Publication No. 2017-60078 There is no mention of measures to prevent motion sickness caused by such videos.

[0456] Therefore, in this embodiment, even when the user moves their face quickly when the shooting unit 40 is capturing images, the resulting image is controlled so that it does not contain fast-changing video scenes, thereby providing a camera system that prevents viewers from experiencing motion sickness.

[0457] Furthermore, for the camera system configuration of Example 3, components identical to those of the camera systems in Examples 1 and 2 will be given the same reference numerals, and redundant explanations will be omitted. Different components will be described in detail as needed.

[0458] As explained using Figures 8H to 8K and Figures 10(b) to (d), the user's face rotates in the up, down, left, and right directions when the imaging unit 40 takes an image.

[0459] Therefore, below, the direction and speed of the user's face movement will be expressed by angular velocity ω, and the distance moved will be expressed by angle Θ.

[0460] The angular velocity ω is calculated by dividing the angle Θ detected by the face direction detection unit 20 by the detection interval.

[0461] Here, examples of actions in which people quickly move their heads include turning around, glancing, and observing moving objects.

[0462] Turning around is the action of quickly turning your head, for example, when a loud noise occurs behind you.

[0463] A quick glance is the action of noticing something in your field of vision, looking at it, but then returning your face to its original position because you're not particularly interested.

[0464] Observing moving objects involves actions such as observing birds flying freely in the sky or kites being flown.

[0465] If these actions occur during imaging by the camera unit 40, and the image in the direction the user's face is facing is simply extracted and developed using the recording range development process, as described above, viewers of the resulting image may experience motion sickness.

[0466] Therefore, the overall control CPU 101 determines that if an angular velocity ω that is greater than or equal to the threshold ω0 is calculated for a predetermined time or longer (a first predetermined time or longer), the user has performed an action of quickly moving their face (turning around, glancing, or observing a moving object). Furthermore, if the overall control CPU 101 determines that the action that occurred is not a glancing or observing a moving object using the method described later with reference to Figure 25, it determines that the action is turning around. In this case, the overall control CPU 101 does not extract the image in the direction the user's face is facing directly using the recording range development process, but rather performs delayed extraction, which extracts the image with a delay relative to the movement of the user's face.

[0467] In this embodiment, the threshold ω0 is set to π / 8 rad / s. This is the speed at which a face moves from 0 degrees forward to 90 degrees to the side in 4 seconds. However, the threshold ω0 is not limited to π / 8 rad / s. For example, based on the frame rate n fps, the threshold ω0 may be set to (n × π) / x rad / s (where x is any value).

[0468] The angular velocity ω is the angle Θ obtained from the image of frame n in this case. n and the time of acquisition t n And the angle Θn-1 obtained from the image of the previous frame n-1 and the time t when it was obtained. n-1 Therefore, angular velocity ω n This can be calculated using the following formula. ω n =(Θ n -Θn-1 ) / (t n -t n-1 ) However, angular velocity ω is ω n-x ~ω n It can also be used as an average of the sums.

[0469] Furthermore, in this embodiment, the predetermined time is set to 0.2 seconds, but this value is not limited to that.

[0470] The following explanation of delayed data extraction when the user is reviewing data will be used with reference to Figure 24.

[0471] In Example 1, Figures 11 and 12 were explained taking into account the distortion of the imaging lens 16, but in this example, for the sake of simplicity, the distortion of the imaging lens 16 is not considered. Also, the calibration process in Example 2 is performed on the frame image, and the center of the image in each frame is assumed to coincide with the center of the user's field of view at the time the image is captured. Furthermore, to explain the case where the face is turned directly to the side, an example is given in which light rays up to a maximum FOV angle of approximately 192° are projected onto the solid-state image sensor 42.

[0472] Region 4000 represents the image-capable pixel area of ​​the solid-state image sensor 42.

[0473] Images 4001 (Figure 24(a)) and 4022 (Figure 24(c)) are frames f cropped with a field of view of 125, aiming at the direction the face is currently facing. n This is an image.

[0474] Images 4002 (Figure 24(a)) and 4021 (Figure 24(c)) are frames f, which were cropped with a field of view of 125, aiming at the direction the face was facing last time. n-1 This is an image.

[0475] Distance 4010 (Figure 24(a)) corresponds to frame f n-1 From the center of image 4002, frame f n This is the distance to the center of image 4001. Below, frame f n-1 From the center of the image to frame fn Let d be the distance to the center of the image.

[0476] Image 4003 (Figure 24(b)) is based on the face direction detected by the face direction detection unit 20. If the angular velocity ω of the face is greater than or equal to the threshold ω0, the delayed extraction frame f' n This image is extracted from the video projected onto area 4000.

[0477] Distance 4011 is frame f n-1 Delayed crop frame f' from the center of image 4002 n This is the distance to the center of image 4003. Below, frame f n-1 Delayed crop frame f' from the center of the image n Let d' be the distance to the center of the image.

[0478] The delay distance of 4012 corresponds to frame f n Frame f' from the center of image 4001 n This is the distance to the center of image 4003, and we will refer to this value as "d" below.

[0479] At this point, comparing the value d at distance 4010 with the value d' at distance 4011, we find that d > d'.

[0480] Next, we will explain an example of how to determine the value of d' using Figure 24(c).

[0481] This section describes the case where the user quickly moves their face 90 degrees to the right from the front (observation direction vo (vector information [0°,0°])). In this case, the frame f where the face is facing forward (observation direction vo (vector information [0°,0°])) n After this is done, and after a short time has passed, the frame f will turn the face 90° to the right from the front. n+x Image 4022 is obtained.

[0482] To prevent motion sickness from video, if the viewer needs to move 90° to the right from the front over a period of t seconds (e.g., 4 seconds) or more, then if the video's frame rate is n fps (e.g., 30 fps), then d'=(f n+x -f n ) / (n fps × t seconds).

[0483] Meanwhile, frame f n From frame f' n As the distance d'' to the point increases, the direction the face is facing is not recorded as the recording direction, so frame f n This means that the subject the user was looking at may not be visible in the image.

[0484] Therefore, the delay time is a predetermined time Th delay If the time exceeds (the second predetermined time), the delayed segmentation will be stopped, and segmentation will be performed in the direction the face is currently facing.

[0485] Here, the delay time refers to the time t0 when the delay began (step S4211 in Figure 26) and the current time t when the face continues to move. n This is the difference in step S4213 (in Figure 26).

[0486] Predetermined value Th delay In this embodiment, it is set to 1 second, but is not limited to this. For example, based on the frame rate n fps, a predetermined value Th delay You may set this to 20 / n seconds. The predetermined value Th delay If the frame rate is 20 / n seconds, the predetermined value Th increases as the frame rate increases. delay This reduces the delay. This is because a higher frame rate reduces the likelihood of motion sickness, allowing the camera to return to the current direction of the face after a short delay.

[0487] On the other hand, if you stop the delayed cut and return to cutting in the direction the face is currently facing, the video scene will switch abruptly. Since such abrupt transitions in video scenes can feel unnatural to the user, you may incorporate fade-outs, fade-ins, or other video effects.

[0488] Additionally, the facial trajectory is preserved so that the cropping process can be resumed for the direction the face is currently facing.

[0489] Here, we will explain the trajectory of the retained face using Figure 25(a) as an example, where it is determined that the user is glancing at the screen.

[0490] As mentioned earlier, the process of stopping the delayed segmentation and returning to segmentation in the direction the face is currently facing occurs when the delay time is a predetermined value Th delay This will be executed if the above conditions are met, but it will also be executed if the person is just glancing around, that is, if their face looks in a specific direction and then immediately turns back to its original direction.

[0491] Figure 25(a) shows an example of the facial trajectory when the user is glancing at the screen.

[0492] Frame f n-3 The center of the image, position 4101, coincides with the user's field of view center when the face begins to move. Subsequently, the user's field of view center is at frame f n-2 ,f n-1 ,f n The image moves to positions 4102, 4103, and 4104, which are the centers of each respective image. Hereafter, this movement of the user's field of view center will be referred to as the face movement vector.

[0493] The facial movement vector then remained at position 4104 for a while before reaching frame f nx+1 ,f nx+2 ,f nx+3 It moves to the center positions of each image, 4105, 4106, and 4107, and frames f nx+3 The image stops at position 4107, which is the center of the image.

[0494] In other words, the facial motion vectors at positions 4101-4104 and positions 4104-4107 are inversely related.

[0495] The overall control CPU 101, when it detects a group of frames in which the facial movement vectors coincide in opposite directions, as illustrated in Figure 25(a), determines that the group of frames is a group of frames in which the user is glancing at the screen.

[0496] In this case, the overall control CPU 101 performs a delayed extraction from the position 4101 where the face began to move until the position 4104 where the face's motion vector began to move in the opposite direction. This is because position 4104 is considered to be the position where the subject the user wanted to glance at is visible.

[0497] On the other hand, after delaying the extraction up to position 4104, the extraction returns to the direction the face is currently facing, that is, the extraction at position 4107 where the face movement has stopped.

[0498] Furthermore, when the user's face is moving, the overall control CPU 101 detects an object located near the center of the face's direction. If the detected object remains in the center of the face's direction, the CPU determines that the user is observing a moving object. In this case, delay extraction is not performed in this embodiment.

[0499] Figure 25(b) shows an example of images for each frame when the user is observing moving objects.

[0500] Frame f n-1 The center of image 4121 coincides with the user's field of view center when the face begins to move. Subsequently, the user's field of view center is at frame f n ,f n+1 ,f n+2 ,f n+3 ,f n+4 It moves to the center position of each of the images 4122-4126.

[0501] The same subject, a bird, remains present near the center of each frame, from image 4121 to 4126.

[0502] When the overall control CPU 101 detects a series of consecutive frames in which the same subject is located near the center of the image, as illustrated in Figure 25(b), it determines that the series of frames is a series of frames in which motion is being observed.

[0503] In this case, the overall control CPU 101 does not perform delayed extraction. This is because performing delayed extraction during motion observation would increase the likelihood that the subject will not be visible in the video.

[0504] Furthermore, if viewers were to view the video, which is extracted from images 4121-4126 in response to the user's rapid facial movements during motion observation, it could potentially cause motion sickness. Therefore, the overall control CPU 101 does not extract images from the frames during motion observation, but instead records the entire pixel area that can be captured by the solid-state image sensor 42, i.e., the entire 4000-area image.

[0505] Furthermore, the threshold ω0, predetermined time, and predetermined value Th mentioned above are also included. delay It may also have a range called a dead zone.

[0506] Next, the motion sickness prevention process according to this embodiment will be explained using the flowchart in Figure 26. This process is executed each time a frame is captured in step S400 while the shooting unit 40 is capturing video.

[0507] In step S4201, the overall control CPU 101 acquires the face direction (observation direction) recorded in the primary memory 103 during the face direction detection process performed for capturing this frame.

[0508] In step S4202, the overall control CPU 101 obtains the position and size (cropping range) of the video recording frame recorded in the primary memory 103 during the recording direction and range determination process performed for this frame capture.

[0509] In step S4203, the overall control CPU 101 (calculation means) calculates the angular velocity ω of the face based on the face direction acquired in step S4201 during the current frame capture, the face direction from the previous frame capture held in the primary memory 103, and the frame rate. Subsequently, the overall control CPU 101 determines whether the face has started moving at an angular velocity ω greater than or equal to the threshold ω0. Specifically, it is determined that the face has started moving at an angular velocity ω greater than or equal to the threshold ω0 if the user's face has moved at an angular velocity ω greater than or equal to the threshold ω0 for a predetermined time (0.2 seconds) or longer. If it is determined that the face has started moving, the process proceeds to step S4204; otherwise, it returns to step S4201. In other words, even if the user's face has moved at an angular velocity ω greater than or equal to the threshold ω0, if the time is less than the predetermined time (less than the first predetermined time), the process returns to step S4201. Furthermore, if the face orientation from the previous frame is not stored in the primary memory 103, and the angular velocity of the face cannot be calculated in step S4203, the process returns to step S4201.

[0510] In step S4204, the overall control CPU 101 determines whether the face has moved by a predetermined angle or more, based on the angular velocity ω of the face calculated in step S4203. If it is determined that the face has moved, the process proceeds to step S4206; otherwise, the process proceeds to step S4205. In addition, in step S4204, it may also be determined whether the face has moved at a predetermined angular velocity or more (YES in step S4203) for a predetermined time (0.2 seconds) or longer.

[0511] In step S4205, the overall control CPU 101 determines whether the movement of the face has stopped based on the angular velocity ω of the face calculated in step S4203. If it is determined that the movement has stopped, the process returns to step S4201; otherwise, the process returns to step S4204.

[0512] In step S4206, the overall control CPU 101 determines whether the subject being imaged is moving (i.e., whether the user is observing a moving object). If it is determined that the subject is moving, the process proceeds to step S4207; otherwise, the process proceeds to step S4208.

[0513] In step S4207, the overall control CPU 101 processes the recording range of the current frame. The logic is to not perform crop development processing, but to perform development processing on the entire RAW data acquired from the entire solid-state image sensor 42, and proceed to step S4205.

[0514] In step S4208, the overall control CPU 101 stores the face direction acquired in step S4201 for the current frame in the primary memory 103 as the face direction for the previous frame, and proceeds to step S4209.

[0515] In step S4209, the overall control CPU 101 (delay means) decides that in the recording range development process for the current frame, it will perform crop development (delayed cropping) on ​​a cropping range centered at a position shifted by a distance d from the face direction of the previous frame. Then, the process proceeds to step S4210.

[0516] In step S4210, the overall control CPU 101 determines whether the start time t0 of the delay time stored in the primary memory 103 has been cleared. If it is determined that it has been cleared, the process proceeds to step S4211; otherwise, the process proceeds to step S4212.

[0517] In step S4211, the overall control CPU 101 stores the current time in the primary memory 103 as the start time t0, and then proceeds to step S4212.

[0518] In step S4212, the overall control CPU 101 determines the delay time based on the angular velocity ω of the face calculated in step S4203, and sets the delay time to a predetermined value Th delay Before reaching the next step, it is determined whether the facial movement has stopped. If it is determined that it has stopped, the process proceeds to step S4215; otherwise, the process proceeds to step S4213.

[0519] In step S4213, the overall control CPU 101 stores the current time as time tn in the primary memory 103 and proceeds to step S4214.

[0520] In step S4214, the overall control CPU 101 calculates the delay time, which is the difference between the time tn stored in the primary memory 103 and the start time t0, and the delay time is set to a predetermined time Th delay Determine whether or not the above is true. A predetermined time Th delay If the above is true, proceed to step S4215; otherwise, return to step S4206.

[0521] In step S4215, the overall control CPU 101 clears the start time t0 stored in the primary memory 103 and proceeds to step S4216.

[0522] In step S4216, the overall control CPU 101 determines the recording direction and field of view in the recording direction / field of view determination unit 30 based on the face direction detected by the face direction detection unit 20, and then proceeds to step S4217.

[0523] In step S4217, the overall control CPU 101 records the metadata flags for the current frame and returns to step S4201. The metadata flags set here are used to determine the timing for applying video effects such as fade-in and fade-out (fade effects) during the secondary recording process described in step S1000 of Example 1.

[0524] In this embodiment, when the angular velocity ω of the face exceeds the threshold ω0, instead of simply extracting the frame in the direction the face is facing, the frame is extracted according to the movement of the face, which has the effect of reducing motion sickness.

[0525] (Example 4) In Example 4, a method for correcting the video cropping range according to the speed of movement of the user's face direction will be explained using Figures 27 and 28.

[0526] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 4, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0527] First, let's explain how a person changes their direction of observation. Typically, when a person finds something of interest at the edge of their field of vision, away from the center, and turns their direction of observation towards it, their face moves first, and beyond a certain point, their body follows with a delay.

[0528] In other words, in such cases, the direction of the imaging lens 16 on the imaging / detection unit 10 (Figure 10(a)) located in front of the clavicle does not move when only the face is initially turned. Subsequently, when the user begins to turn their entire body, the direction of the imaging lens 16 on the camera body 1 also moves. The following explanation will take these characteristics of human body movement into consideration.

[0529] Furthermore, when the face direction detection unit 20 detects the face direction, variations occur due to detection errors. If the video cropping position is calculated based on the face direction detection result which includes such variations, the video recorded secondarily in step S1000 will have a blur similar to camera shake in a typical video, resulting in a poor appearance. Therefore, in order to correct for the fine detection fluctuations, a low-pass filter is applied to the face direction detection result to remove the fine variations.

[0530] Furthermore, if the face direction is detected by tracking even momentary facial movements, such as checking left and right while walking on a public road, the video recorded secondarily in step S1000 will be prone to causing motion sickness. Therefore, in this embodiment, processing is performed to remove (smooth out) even minute facial movement components detected by tracking momentary facial movements of about 1 to 2 seconds. This makes the video recorded secondarily in step S1000 a more visually appealing video.

[0531] Next, an overview of the cropping range correction process in this embodiment will be explained using Figure 27.

[0532] In the graphs shown in Figure 27, the horizontal axis represents time, and the vertical axis represents the angles of the actual observation center (Figure 27(a)), face direction (Figures 27(b), (c)), direction of the imaging lens 16 (Figure 27(d)), and cropping position (Figures 27(e), (f)), respectively. Note that the upward direction on the vertical axis indicates the rightward direction.

[0533] Figure 27(a) is a graph showing the actual movement of the observation center. Note that the angle on the vertical axis in Figure 27(a) does not represent the angle indicating the face direction detected by the face direction detection unit 20, but rather the position of the user's face relative to a fixed position such as the ground (ground reference). In other words, the graph in Figure 27(a) shows that the user is initially facing forward, but begins to turn to the right around 1 second.

[0534] Figure 27(b) is a graph showing the detection result (observation direction vi) of the face direction detection unit 20. The line showing the detection result in Figure 27(b) is not smooth because, as mentioned above, there is variability in the detection result due to detection error. Therefore, in this embodiment, a low-pass filter is applied to the detection result of the face direction detection unit 20.

[0535] Additionally, although not detected in Figure 27(b), the system also removes (smooths) the face direction detected by tracking momentary face movements.

[0536] Figure 27(c) is a graph showing the result of smoothing the detection result of the face direction detection unit 20 in Figure 27(b) by applying a low-pass filter. As shown in Figure 27(c), the line showing the detection result in Figure 27(b) becomes a smooth line when a low-pass filter is applied. However, by applying such a filter, in Figure 27(c), the movement of the detected face direction from the front to the right begins at around 2 seconds, resulting in a delay (time lag) compared to Figure 27(b), where the movement begins almost simultaneously with the case in Figure 27(a). Note that the angle on the vertical axis in Figures 27(b) and (c) is the angle from the direction of the imaging lens 16 (camera body 1 reference), unlike the ground reference in Figure 27(a).

[0537] Furthermore, in Figure 27(b), the tilt becomes gentler from around 4 seconds compared to Figure 27(a). This means that, as shown in Figure 27(d), the camera body 1 (direction of the imaging lens 16) began to move together with the user's body from around 4 seconds, and therefore the relative speed of movement in the face direction detected by the face direction detection unit 20 slowed down.

[0538] Therefore, conventionally, the position obtained by adding the amount of camera body movement (Figure 27(d)) ​​to the face direction detection result smoothed by applying a low-pass filter (Figure 27(c)) was calculated as the cropping position shown in Figure 27(e), i.e., the observation direction that is the center of the target field of view 125. However, when the cropping position is calculated using the conventional method, the cropping position does not follow the actual movement of the observation center, and the resulting video from the secondary recording appears as if the panning suddenly accelerates from around 4.5 seconds when the body movement begins.

[0539] In other words, to eliminate the feeling of incongruity with the actual movement of the observation center, it is preferable to calculate the cropping position (expected value) so that the panning is approximately constant, as shown in Figure 27(f).

[0540] Therefore, in this embodiment, the cutting position is calculated so that the panning does not appear to accelerate suddenly, as shown in Figure 27(e). Note that the movement speed of the cutting position is 0° / second, as shown in Figure 27. In the case of the two types, 10° / second, the face direction detection result in Figure 27(c) is earlier by the amount of the above time lag. By adding the amount of movement of the camera body 1 in Figure 27(d) (1 second earlier in this embodiment), the expected value in Figure 27(f) can be calculated. However, in reality, the movement speed of the cropping position is only of the two types described above; that is, the observation direction does not suddenly accelerate or stop, but decelerates slowly. However, the above calculation method does not allow the expected value to draw a slow deceleration curve. Therefore, in this embodiment, when the movement of the camera body 1 stops, the movement speed of the cropping position from the start of movement in the observation direction until now, or for a certain period in the past, is allocated to several frames so that the expected value draws a deceleration curve.

[0541] The following describes the cropping range correction process in this embodiment step by step, using the flowchart in Figure 28.

[0542] In the following explanation, we will simplify or omit explanations of parts that are common to Examples 1-3 described above.

[0543] Figure 28(a) is a flowchart of the subroutine for determining the recording direction and range in step S300 of Figure 7A according to this embodiment.

[0544] In step S4000a, the observation direction vi obtained in the face direction detection process of step S200 is smoothed using a low-pass filter (smoothing means). As described above using Figure 27(b), the observation direction vi has some variation due to detection errors. A simple method for the low-pass filter is to take a simple moving average of the past several times, for example, 5 to 10 times. However, in this case, the more times the average is taken, the slower the tracking becomes when the direction of the face moves. Also, in cases where the user turns right and then immediately turns left, the user turns furthest to the right. There is also the problem that the observation direction vi cannot be fully detected.

[0545] Furthermore, since the degree of detection error varies depending on the detection method, it is preferable to adjust it accordingly. It is also possible to apply the low-pass filter differently in the vertical and horizontal directions.

[0546] Furthermore, as mentioned above, momentary facial movements are often inconsistent with the purpose of this invention, which is to record the user's experience as video. For example, this includes situations where the face is moved involuntarily to check for safety to the left and right while walking. Therefore, in this embodiment, the observation direction vi obtained when the user returns to approximately the original direction in about 2 seconds is also smoothed in step S4000a.

[0547] Furthermore, while safety checks are often necessary in the left-right and downward directions, they are less necessary in the upward direction, so it may be acceptable to omit the application of a low-pass filter in the upward direction.

[0548] Once the cutting range is determined by the processes from steps S301 to S304 (Figure 7D), the process proceeds to step S4000, where the overall control CPU 101 (second calibration means) executes the cutting range correction process.

[0549] After that, the corrected cropping range is recorded in step S305, and then the subroutine is exited. The cropping range correction process will be explained using the flowchart in Figure 28(b).

[0550] Figure 28(b) is a flowchart of the cropping range correction process in step S4000.

[0551] In Figure 28(b), first, in step S4001, the overall control CPU 101 (movement speed calculation means) acquires gyro information from the angular velocity sensor 107, that is, the movement of the camera body 1 in the current frame (gyro movement amount).

[0552] In this embodiment, an angular velocity sensor 107 was used, but the method is not limited to this as long as the movement of the camera body 1 can be detected. For example, a magnetic sensor that measures the magnitude and direction of a magnetic field (not shown) may be used, or an acceleration sensor 108 that detects acceleration may be used. Furthermore, a method may be used in which a movement vector is detected by extracting feature points and calculating how much those feature points have moved, and the amount of movement of the camera body 1 is calculated. Known methods can be used for extracting feature points. For example, a bandpass filter can be applied to an image from which only the brightness information of two images has been extracted to extract edges, and the amount of movement can be calculated by subtracting multiple edge images with a shift in position and calculating the position where the difference is smallest. This method increases the computational load, but it is one of the preferred methods because it eliminates the need for hardware such as the angular velocity sensor 107, thus allowing for a lighter camera body 1.

[0553] The following explanation continues using the example of acquiring gyro information from the angular velocity sensor 107.

[0554] In step S4002, the movement speed of the camera body 1 (gyro movement speed) is calculated from the gyro information acquired in step S4001 and previously acquired gyro information.

[0555] In step S4003, it is determined whether the gyro movement speed calculated in step S4002 is decreasing. If the movement speed is not decreasing (NO in step S4003), proceed to step S4004; otherwise, proceed to step S4006.

[0556] In step S4004, the overall control CPU 101 (second calibration means / observation direction correction means) calculates the movement speed of the cropping position from the cropping position determined in step S304 and the cropping position acquired in the past. Next, it obtains a subtraction amount by subtracting the previously acquired gyro movement speed, which is the amount of time lag caused by applying a low-pass filter, from the calculated movement speed of the cropping position.

[0557] In step S4005, the movement speed and subtraction amount of the cutting position obtained in step S4004 are stored in the primary memory 103, and the subroutine is exited.

[0558] In step S4006, the expected value is calculated by allocating the sum of the subtraction amounts stored in the primary memory 103 so that the change in the movement speed of each cropping position over a certain past period, also stored in the primary memory 103, remains constant, and then the subroutine is exited. The certain past period may be the period from when the cropping position actually started moving until the present, or the period from when the angular velocity sensor 107 detected the movement of the camera body 1 until the present. Alternatively, to simplify the process, it may be set to a fixed period of approximately 0.5 to 3 seconds. The expected value for periods prior to the above certain past period is set to the movement speed of the cropping position obtained in step S4004.

[0559] Table 1 below shows the displacement (velocity) of the data graphed in Figures 27(a) to (f). Specifically, the movement velocity of the cutting position determined in step S304 is shown in Table 1(c), and the gyro movement velocity calculated in step S4002 is shown in Table 1(d). In addition, the expected value calculated in step S4006 is shown in Table 1(e).

[0560] [Table 1]

[0561] The subroutine for cropping range correction shown in Figure 28(b) will be explained using the example of a user who initially remains still facing forward and gradually looks to the right, as shown in Table 1.

[0562] Initially, the user is looking forward, so the gyro movement speed calculated in step S4002 is approximately 0° / second. In other words, in step S4003, it is determined that the gyro movement speed has not decreased, and the process proceeds to step S4004. In this case, the position of the face also does not change, so the movement speed of the cropping position is also 0° / second. Furthermore, the subtraction amount calculated in step S4004 is also 0° / second.

[0563] After about 1 second, the user begins to turn to the right, but due to the time lag caused by the low-pass filter, the movement speed of the cropping position is still 0° / second, as shown in Figure 27(c). On the other hand, as shown in Figure 27(d), the camera body 1 has not yet moved, meaning the gyro movement speed is also approximately 0° / second. Therefore, just as when the user is still stationary in front, the subtraction amount calculated in step S4004 is also 0° / second.

[0564] As the user turns further to the right, and after about 2 seconds, the movement speed of the cropping position becomes 10° / second, as shown in Figure 27(c). On the other hand, as shown in Figure 27(d), the camera body 1 has not yet moved, meaning the gyro movement speed is still approximately 0° / second. Therefore, the subtraction amount calculated in step S4004 is 10° / second.

[0565] As the user turns further to the right, after about 4 seconds, the user's body also begins to turn to the right. That is, as shown in Figure 27(d), the orientation of the camera body 1 changes, so the gyro movement speed becomes 10° / second. As the body begins to rotate, as shown in Figure 27(b), the actual angular velocity of the face decreases by the relative velocity between the camera body 1 and the face. However, due to the time lag caused by the low-pass filter, the movement speed of the cropped position shown in Figure 27(c) is still 10° / second at this point. Therefore, taking this time lag into account, the subtraction amount calculated in step S4004 is 10° / second.

[0566] As the user turns further to the right, the gyro movement speed remains at 10° / second (Figure 27(d)) ​​from about 5 seconds onward, but the movement speed of the cutting position shown in Figure 27(c) decelerates to 0° / second. Therefore, the subtraction amount calculated in step S4004 is -10° / second.

[0567] Although not shown in Figure 27, when the user finishes turning to the right after about 6 seconds, the gyro movement speed becomes 0° / second, and only then does the process proceed to step S4006. In this case, the total subtraction amounts calculated so far and stored in the primary memory 103 becomes +10° / second. This total subtraction amount is allocated to the primary memory 103 so that the change in movement speed of each cutout position over a certain period in the past remains constant, and the expected value is calculated. Here, the movement speed of the cutout position shown in Figure 27(c) is 10° / second, 10° / second, 10° / second, and 0° / second from the start of acceleration to the present (2 to 6 seconds), as shown in Table 1. Therefore, in order to keep the change in movement speed of each cutout position constant (no change in this case), the expected value for the period from 2 to 6 seconds is set to 10° / second for all of them.

[0568] In this embodiment, explanations were given at one-second intervals for simplicity, but typically, the frame rate for video capture is 24-60 fps. On the other hand, face direction detection and gyroscope detection often do not need to be performed 60 times per second, so it is preferable to change the timing of face direction detection processing and cropping range correction processing from the timing of image capture. For example, image capture may be performed at 60 fps, but the timing of face direction detection processing and cropping range correction processing can be performed at 10 fps without any problems, and can be changed as appropriate considering the application and power consumption.

[0569] As explained above, this embodiment demonstrates how to keep the amount of movement in the observation direction constant, so that when the observation direction changes significantly, the movement of the face and the movement of the body (camera body) combine, preventing a change in the speed of eye movement in the video and resulting in an unappealing video.

[0570] In this embodiment, an example of cropping an ultra-wide-angle image according to the observation direction is shown, but the embodiment is not limited to this. For example, the overall control CPU 101 (imaging direction changing means) may change the imaging direction of the imaging unit 40 according to the observation direction. However, in this case, the camera body 1 must be provided with a mechanism (driving means) that mechanically drives the imaging direction of the imaging unit 40, specifically the orientation of the imaging lens 16 and the solid-state image sensor 42, in the yaw and pitch directions in a manner not shown.

[0571] Furthermore, although this embodiment demonstrates the smoothing of the face direction detection results, it is preferable to perform similar processing when the overall control CPU 101 (vibration isolation means) performs the vibration isolation control described in Embodiment 1, as this also causes a delay in tracking the face direction.

[0572] (Example 5) In Example 5, a method for reducing the difference between the user's field of view and the secondary recorded video (hereinafter referred to as "recorded video") caused by parallax misalignment due to the positional difference between the user's eye position and the mounting position of the shooting / detection unit 10 will be explained using Figures 29 to 34.

[0573] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 5, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0574] First, to aid understanding, we will explain the difference between the user's field of view and the recorded video that occurs in Example 1.

[0575] Figure 29 is a schematic diagram illustrating the relationship between the user's field of view and the target field of view in Example 1, when the observation target 5020 is a close-range subject.

[0576] Figure 29(a) is a schematic diagram showing the image 5900 including the object of observation 5020 as captured by the solid-state image sensor 42, and Figure 29(b) is a schematic diagram showing the positional relationship between the user 5010 and the object of observation 5020.

[0577] As shown in Figure 29(b), when the object of observation 5020 is below the height of the user's eyes 5011, the user's face direction 5015 is directed downwards. At this time, the object of observation 5020 with a background such as a floor (not shown) is visible in the user's field of view.

[0578] In Example 1, the observation direction 5040 (Figure 29(b)) parallel to the user's face direction 5015 detected by the face direction detection unit 20 is set as the recording direction. Therefore, as shown in Figure 29(b), if the observation target 5020 is a close-range subject, a problem arises in that the target field of view 5045 will be set to an area that does not include the observation target 5020.

[0579] In such cases, although the background visible to the user 5010 (such as the floor, not shown) will be different from the background (such as the ceiling, not shown), the recording direction should be set to the direction 5030 which is the target field of view 5035 that includes the object of observation 5020, rather than the observation direction 5040.

[0580] The above problem is caused by parallax misalignment due to the positional difference between the position of the user's eyes 5011 and the mounting position of the shooting / detection unit 10. Therefore, in this embodiment, parallax correction processing is performed to appropriately adjust the recording direction, which is set based on the user's face orientation, according to the parallax misalignment.

[0581] Figure 31 is a block diagram showing the hardware configuration of the camera body 1 according to this embodiment.

[0582] The hardware configuration of the camera body 1 in this embodiment differs from the camera body 1 of Embodiment 1 shown in Figure 5 only in the presence of a distance measuring sensor 5100. While there are no particular limitations on the placement of the distance measuring sensor 5100 in the camera body 1, in this embodiment, as shown in Figure 30, the distance measuring sensor 5100 is provided on the outer edge of the stop switch 15.

[0583] The distance measuring sensor 5100 is a sensor that measures the distance to an object. The configuration of the distance measuring sensor 5100 is not particularly limited. For example, the distance measuring sensor 5100 may be an active type sensor that projects infrared light, laser light, millimeter waves, etc., onto an object and measures the distance to the object by its reflection. Alternatively, the distance measuring sensor 5100 may be a passive type sensor that measures the distance to an object based on the phase difference of light rays transmitted through the imaging lens 16.

[0584] The distance measuring sensor 5100 is connected to the overall control CPU 101 and controlled by the overall control CPU 101.

[0585] Figure 32 is a schematic diagram illustrating the relationship between the user, the calibrator 850, and the target field of view 5080 during calibration, including parallax correction processing, in this embodiment.

[0586] Figure 32(a) is a schematic diagram showing the image 5900 including the calibrator 850 as projected onto the solid-state image sensor 42, and Figure 32(b) is a schematic diagram showing the positional relationship between the user 5010 and the calibrator 850.

[0587] The target field of view 5080 in Figure 32(a) is the target field of view when calibration, including the parallax correction process described later, has not been performed, and the face direction 5015 detected by the face direction detection unit 20 is facing forward.

[0588] On the other hand, the target field of view 5090 in Figure 32(a) is the target field of view when calibration including the parallax correction process described later has been performed and the face direction 5015 detected by the face direction detection unit 20 is facing forward.

[0589] Figure 33A is a flowchart of the parallax correction mode processing, which is part of the preparation process for step S100 in Figure 7A in this embodiment. The details of this process will be explained below using Figures 32(a) and (b).

[0590] In the preparation operation process of step S100 in Figure 7A related to this process, when the parallax correction mode is activated by the user 5010's operation on the calibrator 850 (step S5101), the display device control unit 801 displays the positioning index 851 (step S5102).

[0591] Next, the display control unit 801 instructs the user to position (specified position) where the calibrator 850 should be held. Specifically, the display control unit 801 instructs the user 5010 to hold the positioning indicator 851 in front of them at eye level by displaying an instruction 855 similar to that in Figure 22A (step S5103).

[0592] When the instruction display 855 is shown, the user 5010 holds the calibrator 850 over the designated position indicated in step S5103 and directs their face direction 5015 towards the positioning indicator 851 (front). At this time, the user 5010, the positioning indicator 851, and the imaging / detection unit 10 are in the positional relationship shown in Figure 32(b).

[0593] Subsequently, when the display device control unit 801 determines that the user is viewing the positioning indicator center 852 in the front of their field of view, it measures the distance 5050 (Figure 32(b)) between the imaging / detection unit 10 and the positioning indicator 851 using the distance measuring sensor 5100 (step S5104).

[0594] Next, the overall control CPU 101 detects the horizontal axis 5060 of the imaging and detection unit 10 using the angular velocity sensor 107 (attitude detection means) (step S5105). This identifies the horizontal position 5065 of the image 5900 (Figure 32(a)) captured by the solid-state image sensor 42.

[0595] Furthermore, in step S5105, the overall control CPU 101 obtains the distance 5855 (Figure 32(a)) between the center of the positioning index 851 on the image 5900 and the horizontal position 5065. Subsequently, the overall control CPU 101 (angle calculation means) calculates the angle 5055 (Figure 32(b)) between the horizontal axis 5060 and the direction of the positioning index 851 as seen from the imaging / detection unit 10. This calculation is performed using the distance 5855 and information stored in memory (e.g., built-in non-volatile memory 102) regarding the angle from which light, which is imaged at a certain point on the image 5900, enters.

[0596] Subsequently, the overall control CPU 101 (vertical distance calculation means) uses the distance 5050 and the angle 5055 calculated in step S5105 to calculate the vertical distance 5070 between the shooting / detection unit 10 and the user's eye 5011 (step S5106), and then exits this subroutine.

[0597] Here, a method for measuring the vertical distance 5070 between the imaging / detection unit 10 and the user's eye 5011 is described using a method different from that of Example 2, but the method is not limited to this. For example, the vertical distance 5070 between the imaging / detection unit 10 and the user's eye 5011 may be measured using the method described in Example 2, or the user 5010 may be directly asked to input the value of the vertical distance 5070.

[0598] The calibration process, including the parallax correction process, in this embodiment is basically the same as the process in steps S3101 to S3111 in Figure 21 that is performed in Embodiment 2, so its explanation is omitted.

[0599] However, in step S3110, in addition to the processing described in Example 2, parallax correction is performed based on the vertical distance 5070 (Figure 32(b)) calculated in the parallax correction mode processing in Figure 33A. That is, calibration is performed so that the user's field of view 5010 and the target field of view 125 coincide at infinity.

[0600] Figure 33B is a flowchart of the recording direction and range determination subroutine S300 described in Figure 7A in this embodiment. This process will be described below with reference to Figure 34. Steps in Figure 33B that overlap with those in Figure 7D are denoted by the same reference numerals, and redundant explanations are omitted.

[0601] In Figure 33B, first, the overall control CPU 101 acquires distance information of the image capture range (imaging area) using the distance measuring sensor 5100 (distance measuring means) (step S5301).

[0602] Next, the overall control CPU 101 (creation means) creates a defocus map 5950 (Figure 34(a); distance map information) based on the distance information (measurement result by the distance measuring sensor 5100) obtained in step S5301 (step S5302).

[0603] The defocus map 5950 in Figure 34(a) represents the defocus map created when imaging was performed with the observation target 5020 floating in the room, as shown in Figure 34(c). Here, in order to clearly show the distance information in the defocus map 5950, it is represented in six distance areas (1) to (6) from the closest distance from the imaging / detection unit 10. However, in reality, the defocus map 5950 could also be created without any steps.

[0604] Next, the overall control CPU 101 calculates the direction of the observation target 5020 as seen from the imaging / detection unit 10 based on the defocus map 5950, face direction 5015, and vertical distance 5070 (Figure 32(b)) (step S5303). In other words, parallax correction is performed for the observation direction set based on the face direction.

[0605] After that, the process in steps S301 to S305 in Figure 7D is performed, and then the subroutine is exited.

[0606] By using the defocus map 5950 created in this way and the detection result of the face direction 5015, it is possible to calculate the direction of the observation target 5020 as seen from the shooting / detection unit 10. However, due to the parallax shift explained using Figure 29, it is not possible to measure only the distance to the observation target 5020 with the distance measuring sensor 5100 without creating the defocus map 5950.

[0607] The degree of parallax shift described in this embodiment depends on the distance between the user 5010 and the object being observed. That's different. In other words, for an object of observation that is a certain distance away from the user 5010, the effect of parallax shift can be ignored, so in the recording direction and range determination process in Example 1, it is possible to cut out and record the image with a target field of view that includes the object of observation. For example, when the user 5010 observes the object of observation 5021 (Figure 34(c)), the object of observation 5021 is located in the medium-distance area (5) which is a certain distance or more away from the user 5010, so it is not necessary to perform parallax correction of the recording direction in step S5303. This is because the target field of view 5043 (Figure 34(b)), which is set according to the recording direction 5041 (observation direction) inferred based on the face direction 5016 detected by the face direction detection unit 20, also includes the object of observation 5021.

[0608] On the other hand, according to this embodiment, the distance range between the user and the object being observed, in which the user 5010 can keep the object within the target field of view, can be expanded to a closer distance than in Embodiment 1. For example, suppose the user 5010 is observing an object being observed 5020 (Figure 34(a)) located in a close-range area (1) that is close to the user 5010. In this case, in Embodiment 1, the observation direction 5040 (recording direction) is inferred based on the face direction 5015 detected by the face direction detection unit 20. The target field of view 5042 (Figure 34(b)) set according to this observation direction 5040 does not include the object being observed 5020. However, in this embodiment, parallax correction is performed for the observation direction 5040 in step S5303 of Figure 33B, and the target field of view 5036 including the object being observed 5020 is set according to the recording direction after this parallax correction. Therefore, it becomes possible to capture images of observation targets, such as observation target 5020, where the distance from the user 5010 is so close that the effect of parallax shift cannot be ignored.

[0609] Furthermore, according to this embodiment, it becomes possible to record an observation target located in the medium-range area (5) closer to the center of the target field of view. For example, when the user 5010 is observing an observation target 5021 (Figure 34(a)) located in the medium-range area (5), if parallax correction of the recording direction 5041 is not performed as in Embodiment 1, the target field of view 5043 will be set with the observation target 5021 at the top. On the other hand, in this embodiment, parallax correction for the recording direction 5041 is performed in step S5303 of Figure 33B, and according to the recording direction after this parallax correction, a recording area 5037 is generated in which the observation target 5021 is at the center.

[0610] As shown above, by implementing the parallax correction in this embodiment, it becomes possible to capture the object of observation more precisely in the center of the cropped image compared to Embodiment 1.

[0611] In this embodiment, parallax correction is also performed during calibration to ensure that the user's field of view and the recording area coincide at infinity. Then, parallax correction is performed such that the closer the distance between the user and the observed object during imaging, the greater the deviation in the recording direction before and after correction. However, the parallax correction of this embodiment may also be performed for finite positions, for example, subjects at a distance greater than the position of the calibrator 850 relative to the user in the calibration process of Embodiment 2, or subjects at a close distance.

[0612] (Example 6) In Example 6, the method for determining the cutting range when the calculation of the observation direction fails will be explained using Figures 35, 36A, and 36B.

[0613] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 6, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0614] In Example 1, as shown in Figure 7A, the recording direction and range are determined in step S300 based on the observation direction calculated from the face direction detected by the face direction detection unit 20 in step S200. The target field of view is set during processing. However, the face direction detection unit 20 may be covered by obstacles such as collars or hair, the face direction detection unit 20 may malfunction, or the face direction detection unit 20 may move away from the user. In such cases, the user's face direction cannot be acquired. In such cases, it becomes impossible to capture images of the target field of view that the user wanted to capture.

[0615] Japanese Patent Publication No. 2007-74033 If the second camera, which is capturing images of the user, fails to detect the user, the failure to detect the user is not saved in the history of previous user observation information, and the user detection is attempted again. Also, when capturing images by tracking the direction of the face, if face direction detection fails, the imaging direction is determined according to the situation, so that images are captured that do not deviate significantly from the user's intentions.

[0616] In contrast, in this embodiment, if the user's face direction can be detected, the face direction detection unit 20 detects the face direction, as in Embodiment 1, and captures an image of the target field of view in the recording direction based on the observation direction calculated based on this detection. On the other hand, if the user's face direction cannot be detected and the user's observation direction cannot be calculated, an image of the target field of view that takes the user's intention into account is captured. That is, in this embodiment, when the face direction detection process is completed in step S200, the observation direction determination process is executed before the recording direction / range determination process is executed in step S300. In this process, if the face direction detection unit 20 fails to detect the user's face direction, the user's intention is determined according to the situation, and the observation direction is estimated. That is, an image of the target field of view in the recording direction is captured based on information other than the observation direction calculated from the face direction.

[0617] Figure 35 is a flowchart of the observation direction determination process according to this embodiment, which is executed by the overall control CPU 101. This process will be explained below using Figures 36A and 36B.

[0618] First, in step S6001, the face direction detection unit 20 determines whether it has been able to acquire the face direction. If the face direction has been acquired, the process proceeds to step S6004, where the overall control CPU 101 (mode transition means) switches the mode of this process to the face direction mode (first imaging mode) and determines the observation direction calculated from the face direction using the method shown in Example 1 as the recording direction. After that, the process exits this subroutine.

[0619] On the other hand, if the face direction cannot be obtained (NO in step S6001), the overall control CPU 101 (mode transition means) proceeds to step S6002 to transition to another mode and determines whether or not there is a subject that has been tracked in the past.

[0620] Here, we will explain the determination process in step S6002 using Figure 36A, which shows the relationship between the detection state of the user's observation direction for each frame and the captured video.

[0621] In Figure 36A, n is the frame number of the video, θ is the horizontal movement angle of the user's face, and User State indicates the positional relationship between the user and the object of observation in each frame. The overall video shows the ultra-wide-angle video captured by the camera unit 40 in each frame, and the captured video shows the image secondarily recorded in each frame.

[0622] Figure 36A illustrates a case where, as shown in each screen of the user state, the user is observing an object indicated by a "□" at the bottom of the screen, and the user's object could not be detected in n=5, or the 5th frame.

[0623] In this embodiment, the current frame is used as the reference point, and the four frames preceding it are defined as a predetermined period. If the same subject is found to be present in the captured video three or more times during this predetermined period, it is determined that the subject was previously tracked.

[0624] As shown in Figure 36A, for n=1 to 4, even though the movement angle θ changes by +10° each time, the captured video contains an object indicated by "□" which can be identified as the same subject. Therefore, for n=5, it is determined that there is a subject that was previously tracked.

[0625] Furthermore, the judgment criteria in step S6002 may be changed in accordance with the face direction detection period and the accuracy of the face direction detection unit 20.

[0626] Returning to Figure 35, if it is determined that there is a subject (the same subject) that was tracked within the past predetermined time (YES in step S6002), proceed to step S6005.

[0627] In step S6005, the processing mode is switched to a pre-subject tracking mode (second imaging mode) where the pre-subject direction is set as the recording direction. After determining the recording direction to track the subject, the process proceeds to step S6008. Thus, in this embodiment, even if the face direction can no longer be detected, if there is a subject that was tracked in the past, the system switches to the pre-subject tracking mode to determine the recording direction, thus reflecting the user's intentions immediately before the recording. Note that the subject recognition method and subject tracking detection method in the captured video by the overall control CPU 101 (subject recognition means) are publicly known, so a detailed explanation is omitted.

[0628] On the other hand, if it is determined that there is no subject that has been tracked in the past (Step S60 02 If NO, proceed to step S6003.

[0629] In step S6003, it is determined whether a subject previously registered in the built-in non-volatile memory (subject registration means) has been detected in the most recent captured video.

[0630] In this embodiment, pre-registration of a subject is performed by the user specifying an image containing the person to be photographed from images stored in the display device 800, the display device control unit 801 recognizing the characteristics of the selected person, and transmitting this information to the overall control CPU 101 in the camera body 1. The subject detected in step S6003 is not limited to this, and may also be a subject included in the captured video acquired at the readout completion timing or other detection timings. Furthermore, whether the pre-registered subject matches the subject in the most recent captured video is determined by a pattern matching method. Since the pattern matching method is publicly known, a detailed explanation is omitted.

[0631] If it is determined that a subject that was registered in advance has been detected in the latest recorded video (YES in step S6003), proceed to step S6006.

[0632] In step S6006, the processing mode is switched to the registered subject mode (third imaging mode), where the direction of the subject detected in step S6003 is used as the recording direction, and the process proceeds to step S6008.

[0633] On the other hand, if it is determined that a pre-registered subject was not detected in the latest captured video (NO in step S6003), it is determined that it was not possible to estimate the subject to be observed, and the process proceeds to step S6007.

[0634] In step S6007, the overall control CPU 101 (angle of view changing means) switches the mode of this process to the subject-lost mode (fourth imaging mode), in which the recording direction before the failure to detect the face direction remains unchanged, while the imaging angle is changed to a wider angle compared to the previous angle of view. Then, the process proceeds to step S6008. Note that in the subject-lost mode, the recording direction may be continuously moved by the amount of change in the observation direction before the failure to detect the face direction.

[0635] Here, we will explain the case where the system proceeds to step S6007, which is in subject loss mode, using Figure 36B.

[0636] Figure 36B illustrates the case where n=5, or the user's object of observation, could not be detected in the 5th frame.

[0637] In the example in Figure 36B, no major subject was found for n=1 to n=4, and the pre-registered subject was not found in the video captured at n=5. Therefore, the observation direction for n=5 is shifted to the rightward direction on the full-screen image, which is the direction of movement for n=1 to n=4, with inertia. In addition, the field of view extracted from the full-screen image is changed to a wide-angle.

[0638] In step S6008, if the recording direction was determined from a direction other than the face direction in any of steps S6005 to S6007, the overall control CPU 101 (notification means) notifies the user of an error (detection error) indicating that face direction detection failed. After that, the subroutine is exited. In this embodiment, a warning is issued to the user using the vibrating body 106 shown in Figure 5. However, the notification method in step S6008 is not limited to the method in this embodiment, and other notification methods may be used, such as a warning using the LED 17 or a display device 800 or other terminal linked to the camera body 1.

[0639] As described above, in this embodiment, if the direction of the face cannot be detected, the recording direction and field of view are changed according to the situation, thus preventing the user from missing the image of the intended field of view that they originally wanted to capture.

[0640] In other words, in this embodiment, if the direction of a face cannot be detected, and a subject that has been tracked in the past or a pre-registered subject can be detected, the camera will track that subject. On the other hand, if such a subject cannot be detected, the field of view is changed to a wider angle than the standard field of view in order to prevent missed shots and to make it easier to re-detect the subject.

[0641] This prevents unintended images from being captured due to a failure in detecting the user's facial orientation.

[0642] Although the processing in steps S6001 to S6008 is performed every frame, the mode can be changed again after transitioning to each mode based on mode determination information, such as whether the face direction has been acquired from the face direction detection unit 20. For example, in this embodiment, if a pre-registered subject is detected as a result of widening the field of view in the subject-lost mode, the system switches to the registered subject tracking mode, which uses the direction of the detected subject as the observation direction. In this case, the widened field of view is returned to the standard field of view.

[0643] Furthermore, although the mode was changed with a single determination in this embodiment, the mode may be switched based on multiple results depending on the frame rate and face direction detection capability.

[0644] (Example 7) In Example 7, a method for determining the observation direction according to the accuracy (reliability) of face direction detection will be explained using Figures 37 to 40.

[0645] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 7, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0646] In Example 6, the mode for determining the observation direction was switched depending on whether or not the face direction was detected, preventing imaging in a recording direction unintended by the user. On the other hand, Japanese Patent Publication No. 2007-74033 If the user's face direction cannot be consistently and reliably detected, as in the example shown, the image may be captured at a field of view unintended by the user. An example of a situation where the face direction cannot be consistently and reliably detected is when the camera body 1's shooting / detection unit 10 is attached to the front of the collarbone, as shown in Figure 1B, where the accuracy of face direction detection may decrease due to the influence of the collar or hair.

[0647] As shown in Figure 37, when the user is facing to the right (Figures 37(b), (c)) compared to when the user is facing forward (Figure 37(a)), a larger area of ​​the chin and cheeks is hidden by the body and shoulders. In other words, depending on the direction of the face, the area of ​​the face that can be used for detecting the direction of the face becomes smaller, and the detection accuracy is likely to decrease. This characteristic of the camera body 1 depends largely on the position in which the user is wearing the camera body 1.

[0648] Therefore, in this embodiment, the accuracy (reliability) of face direction detection is calculated according to the mounting position of the camera body 1 and the detection result of face direction. If the reliability is high, the face direction is reflected more in the observation direction, and if the reliability is low, other information is reflected more in the observation direction. This makes it possible to capture images while understanding the user's intentions.

[0649] Figure 38 is a flowchart of the observation direction determination process when acquiring face direction, according to this embodiment, which is performed instead of the process in step S6004 of Figure 35. This process is executed by the overall control CPU 101 (observation direction determination means).

[0650] First, in step S7001, the overall control CPU 101 (first observation direction calculation means, reliability calculation means) calculates the face direction θ acquired by the face direction detection unit 20 during the imaging of the nth frame. n Face direction reliability T based on (first observation direction) n Calculate.

[0651] Face direction reliability T n It is calculated as follows:

[0652] First, the direction of the face θ n The direction of the face θ yaw ,θ pitch ,θ roll It is divided into three components. Here, the face direction θ yaw θ represents the rotational component of the face moving from side to side, and the face direction θ. pitch This represents the rotational component that moves the face up and down, and the face direction θ. roll This represents a rotational component that moves as if tilting the head.

[0653] In this embodiment, the camera body 1 is attached to the user's clavicle and the direction of the face is detected from below the face, so Tn (0 ≤ Tn ≤ 1) can be calculated using the following equation 701.

[0654]

number

[0655] In this embodiment, the facial direction reliability T is calculated using Equation 701. nAlthough this was calculated, it is also possible to use a weighted average of face direction reliability calculated in the past, weighted according to the face direction detection accuracy of the face direction detection unit 20 and the frame rate of detection. n When calculating the value, factors such as the accuracy of pattern matching and the mounting position may be weighted accordingly.

[0656] Furthermore, in this embodiment, the reliability of the prediction of the subject area was calculated using the coefficient of determination obtained from equation 701. However, the method for calculating the coefficient of determination is not limited to this. For example, The adjusted coefficient of determination, which can be estimated by the calibration in Example 2, may be used according to the mounting position. Furthermore, if the detection accuracy is deemed low during calibration, the coefficient of determination may be changed accordingly. Additionally, when detecting face direction using machine learning, the precision rate may be reflected in the coefficient of determination.

[0657] In step S7002, the overall control CPU 101 calculates the following angular velocity A in the face direction. Specifically, the face direction θ obtained from the face direction detection unit 20 during the acquisition of the nth frame. n and face direction acquisition time t n And the face direction information θ from the previous frame, stored in the primary memory 103. n-1 and the acquisition time t n-1 The angular velocity A is calculated using the following equation 702.

[0658]

number

[0659] In step S7003, the overall control CPU 101 (observation direction prediction means) predicts the current face direction from the past face direction transitions stored in the primary memory 103. In this embodiment, the current frame is used as the reference, and the four frames prior to it are defined as a predetermined period. If the face direction has continuously shifted in a certain direction that can be determined to be the same three or more times during these four frames, it is determined that the observation direction can be predicted from the past face direction and angular velocity. Furthermore, when making this prediction, the predicted angular velocity B, which is the weighted average of the angular velocities obtained from the past four frames, is calculated using the following equation 703, and the predicted face direction θ is determined. ave The (second observation direction) is calculated using equation 704 below. The calculations in equations 703 and 704 correspond to the processes shown in Figures 40(a1) and (a2), respectively.

[0660] Furthermore, the length of the predetermined period used in step S7003 and the method of applying the weighted average may be changed according to the frame rate and the detection accuracy of the face direction detection unit 20.

[0661]

number

[0662] In this embodiment, the internal information used to predict the observation direction in step S7004 is subject detection. Although it was the output history, it is not limited to this. For example, according to the mounting position and performance of the camera body 1, the face information of the user reflected in the imaging unit 40, and the information on the movement and posture of the camera body 1 detected by the angular velocity sensor 107 and the acceleration sensor 108, the prediction of the observation direction may be performed. Also, as in step S6006 of Example 6, when there is a pre-registered subject, the overall control CPU 101 (the third observation direction prediction means) predicts the direction of the pre-registered subject on the latest captured video as the predicted observation direction θ sub and may obtain it.

[0663] In step S7005, the overall control CPU 101 stores the face direction detection-related information as a history in the primary memory 103. Here, the face direction detection-related information includes the angular velocity A of the face direction generated in step S7002, the face direction reliability T calculated in step S7001 n , the face direction θ detected by the face direction detection unit 20 n , the face direction acquisition time t n , and the information indicating the generation time of each of these pieces of information.

[0664] In step S7006, when the face direction reliability T calculated in step S7001 n is greater than or equal to a predetermined value, it is determined that the reliability of the face direction is high, and the process proceeds to step S7009.

[0665] In step S7009, the overall control CPU 101 determines the face direction as the current observation direction θ' n and proceeds to step S7013.

[0666] On the other hand, when the face direction reliability T calculated in step S7001 n is less than the predetermined value (NO in step S7006), the process proceeds to step S7007.

[0667] In step S7007, in step S7003, the predicted face direction θ ave can be estimated, and |θ n -θ aveIf the condition that | is within a predetermined angle is met, proceed to step S7010. In this embodiment, the determination is made using the predetermined angle π / 8.

[0668] In step S7010, the overall control CPU 101 (first observation direction prediction means) determines θ n and θ ave and facial direction reliability T n Using the current observation direction θ' n Determine the current observation direction θ'. In this example, the current observation direction θ' is determined. n ′ is calculated using the following equation 705, and the process proceeds to step S7013. The calculation in equation 705 corresponds to the process shown in Figure 40(b). As shown in Figure 39, the face direction θ yaw The smaller the absolute value of the angle, the lower the face direction reliability T. n As it becomes larger, the current observation direction θ' n As shown in equation 705, the face direction θ n This is reflected more. On the other hand, the face direction θ yaw The larger the absolute value of the angle, the higher the face direction reliability T. n As it becomes smaller, the current observation direction θ' n As shown in equation 705, the face direction θ yaw Other information (predicted face direction θ) ave ) is reflected more.

[0669]

number

[0670] In step S7011, the overall control CPU 101 (second observation direction prediction means) determines the face direction θ n And, the predicted observation direction θ sub And, facial direction reliability Tn Using the current observation direction θ' n Determine the current observation direction θ'. In this example, the current observation direction θ' is determined. n The value is calculated using equation 706 below, and the process proceeds to step S7013. Similar to step S7010, the face direction θ is as shown in Figure 39. yaw The angle of the absolute The smaller the logarithm value, the greater the facial direction reliability T. n As it becomes larger, the current observation direction θ' n As shown in equation 706, the face direction θ n This is reflected more. On the other hand, the face direction θ yaw The larger the absolute value of the angle, the higher the face direction reliability T. n As it becomes smaller, the current observation direction θ' n As shown in equation 706, the face direction θ yaw Other information (predicted observation direction θ) sub ) is reflected more.

[0671]

number

[0672] In step S7012, the previous observation direction θ' n-1 The current observation direction θ' is moved in the observation direction with inertia based on the displacement in the past observation direction. n The camera determines the angle of view and sets it wider than the specified value, then proceeds to step S7013. This reduces the possibility of the user missing the intended subject.

[0673] In this example, facial direction reliability T n And the current observation direction θ' depending on the detection status of the subject. n The calculation method has been changed, but is not limited to this. For example, the predicted face direction θ ave and predicted observation direction θ subWhen calculating each of these, the confidence level (prediction direction confidence level) is also calculated, and the calculated observation direction θ' is determined according to the calculated confidence levels. n You may also try to correct it.

[0674] Furthermore, if each of the calculated confidence values ​​falls below a predetermined value, there is a higher possibility that the user will miss capturing the intended subject. Therefore, it is preferable to set the field of view to a wider angle than the specified field of view. In this case, the process may proceed to step S7012. Subsequently, if any of the calculated confidence values ​​exceed a predetermined value, it is preferable to return the field of view to the specified field of view.

[0675] The processing shown in Figure 38 results in a facial direction reliability T n When the face direction θ is high, n current observation direction θ' n This is the decision. On the other hand, facial direction reliability T n If the facial direction reliability T is low, depending on the situation, n Information from when the value was high, and information other than face direction, are used to determine the current observation direction θ'. n Determine the recording direction, and if necessary, widen the field of view.

[0676] In other words, in this embodiment, the facial direction reliability T n If the value is low and the accuracy of face direction detection is expected to be low, predict the face direction θ ave or predicted observation direction θ sub By using this method, it is possible to prevent the capture of unintended images due to failures in face direction detection.

[0677] (Example 8) In Example 8, a method for mounting the camera body 1 in a stable position will be explained using Figures 41 to 45.

[0678] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 8, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0679] First, let's explain how to adjust the angle of the connection part 80 (neck-hanging mechanism).

[0680] Figure 41 is an enlarged view showing the imaging and detection unit 10 from the side.

[0681] Figure 41(a) shows the connection part 80 in its standard position, Ax0, and Figure 41(b) shows the connection part 80 in position Ax1, which is rotated by an angle θA1 with respect to position Ax0 around the rotation axis OA. Figure 41(c) is a schematic diagram showing the internal mechanical mechanism of the angle holding part 81 that can be seen when the outer casing of the angle holding part 81 is removed.

[0682] As shown in Figure 41(c), an angle adjustment mechanism 8100 (neck-hanging angle adjustment means) is arranged inside the angle holding section 81.

[0683] The angle adjustment mechanism 8100 consists of an angle adjustment cam 8101 that adjusts the angle of the angle holding unit 81 relative to the imaging / detection unit 10, and a locking member 8102 that locks the angle adjustment cam 8101. Note that OA shown in Figure 41(a) is the rotation center of the angle holding unit 81 and is located at the center of the angle adjustment cam 8101.

[0684] The locking member 8102 is biased against the angle adjustment cam 8101 by a spring (not shown), but while the angle adjustment button 85 (Figure 2F) is pressed, the biasing is released and it moves away from the angle adjustment cam 8101. In other words, only while the angle adjustment button 85 is pressed, the angle holding part 81 of the connecting part 80 becomes rotatable relative to the shooting / detection part 10.

[0685] The user can adjust the position of the connection part 80 from position Ax0 (Figure 41(a)) to position Ax1 (Figure 41(b)) by pressing the angle adjustment button 85 and rotating the angle holding part 81 relative to the shooting / detection part 10.

[0686] In this embodiment, a stepped adjustment mechanism using an angle adjustment cam 8101 and a locking member 8102 is used as a mechanism for maintaining the angle of the angle holding unit 81 relative to the imaging / detection unit 10. However, a mechanism that enables stepless adjustment using sliding resistance may also be used.

[0687] Furthermore, in this embodiment, the user rotates the angle holding part 81 while pressing the angle adjustment button 85, but the invention is not limited to this configuration. For example, a balanced configuration is possible in which the angle holding part 81 rotates when an external force exceeding a certain threshold is applied to the angle holding part 81, without requiring the angle adjustment button 85. For example, a ball could be used instead of the locking member 8102, or sliding resistance could be used.

[0688] Figure 42 is a side view showing the camera body 1 attached to the user.

[0689] Figure 42(a) shows a user wearing the camera body 1 with the connection part 80 at position Ax0 and the band part 82 adjusted to be long. Figure 41(b) shows a user wearing the camera body 1 with the connection part 80 at position Ax0 and the band part 82 adjusted to be short. Figure 41(c) shows a user wearing the camera body 1 with the connection part 80 at position Ax1 and the band part 82 adjusted to be short.

[0690] As shown in Figures 42(a) and (c), if the relationship between the position of the connection part 80 and the length of the band part 82 is suitable for the user, the imaging lens 16 will face the user directly. On the other hand, as shown in Figure 42(b), if the relationship between the position of the connection part 80 and the length of the band part 82 is not suitable for the user, the imaging lens 16 will not face the user directly. In this case, Figure 42(b), the optical axis of the imaging lens 16 is facing upward.

[0691] Since the connection part 80 is configured to allow adjustment of its position, the user can adjust the camera so that the optical axis of the imaging lens 16 is approximately parallel to the user's natural line of sight. The main unit 1 can then be attached. Of course, if the appropriate mounting position for the camera body 1 for the user is such that the optical axis of the imaging lens 16 is horizontal, then proper mounting is also possible.

[0692] Next, we will explain how to adjust the angle of the chest connection pad 18.

[0693] Figure 43 is an enlarged view showing the shooting / detection unit 10 from the side when the connection part 80 is hidden.

[0694] Figure 43(a) shows the chest connection pad 18 in its standard position Bx0, and Figure 43(b) shows the chest connection pad 18 in position Bx1, which is rotated by an angle θB1 relative to position Bx0 around the rotation axis OB. Figure 43(c) is a schematic diagram showing the internal mechanical mechanism of the imaging / detection unit 10 that can be seen when the exterior of the imaging / detection unit 10 is removed.

[0695] As shown in Figure 43(c), an angle adjustment mechanism 8200 (grounding angle adjustment means) is located inside the imaging and detection unit 10.

[0696] The angle adjustment mechanism 8200 consists of an angle adjustment cam 8201 for adjusting the angle of the chest connection pad 18 relative to the imaging / detection unit 10, and a locking member 8202 for locking the angle adjustment cam 8201. Note that OB shown in Figure 43(c) is the pivot point of the chest connection pad 18.

[0697] The locking member 8202 is biased against the angle adjustment cam 8201 by a spring (not shown), but while the angle adjustment button 8203 is pressed, the biasing is released and it moves away from the angle adjustment cam 8201. In other words, the chest connection pad 18 becomes rotatable relative to the imaging / detection unit 10 only while the angle adjustment button 8203 is pressed.

[0698] The user can adjust the position of the chest connection pad 18 from position Bx0 to position Bx1 by rotating the chest connection pad 18 relative to the imaging / detection unit 10 while pressing the angle adjustment button 8203.

[0699] In this embodiment, a stepped adjustment mechanism using an angle adjustment cam 8201 and a locking member 8202 is used as a mechanism to maintain the angle of the chest connection pad 18 relative to the imaging / detection unit 10. However, a mechanism that enables stepless adjustment using sliding resistance may also be used.

[0700] Furthermore, in this embodiment, the user rotates the chest connection pad 18 while pressing the angle adjustment button 8203, but the invention is not limited to this configuration. For example, a configuration that does not require the angle adjustment button 8203 and balances the chest connection pad 18 so that it rotates when an external force exceeding a certain threshold is applied to it, such as using a ball instead of the locking member 8202 or using sliding resistance, is also possible.

[0701] Figure 44 is a side view showing the camera body 1 attached by the user, with the connection part 80 hidden.

[0702] Figure 44(a) shows a user with an upright chest wearing the camera body 1 with the chest connection pad 18 in position Bx0. Figure 44(b) shows a user with a reclined chest wearing the camera body 1 with the chest connection pad 18 in position Bx0. Figure 44(c) shows a user with a reclined chest wearing the camera body 1 with the chest connection pad 18 in position Bx1.

[0703] As shown in Figures 44(a) and (c), if the position of the chest connection pad 18 is suitable for the tilt of the user's chest, the chest connection pad 18 will make contact with the user's chest over a wide area. On the other hand, as shown in Figure 44(b), if the position of the chest connection pad 18 is not suitable for the tilt of the user's chest, the chest connection pad 18 will only make contact with the user's chest over a small area. As shown in Figure 44(b), if the area in contact with the user's chest of the chest connection pad 18 becomes small, the shooting / detection unit 10 will easily move relative to the user's body when the user moves, causing the captured image to become significantly blurred.

[0704] Since the chest connection pad 18 is configured to allow for angle adjustment, the user can attach the camera body 1 so that the chest connection pad 18 is positioned over a wide area of ​​their chest, thereby suppressing blur in the captured image.

[0705] In this embodiment, the chest connection pad 18 is located in the imaging / detection unit 10, but the same effect can be obtained even if it is located in the connection unit 80. In this case, for example, a mechanism similar to the angle adjustment mechanism 8100 shown in Figure 41(c) is located inside the connection unit 80 to adjust the angle of the chest connection pad 18 relative to the connection unit 80.

[0706] Next, the configuration of the band section 82 and the electrical cable 84 will be described in detail.

[0707] As described in Example 1, the battery unit 90 (power supply) and the shooting / detection unit 10 of the camera body 1 are separate modules that are electrically connected via an electrical cable 84.

[0708] If the electrical cable 84 and the band portion 82 were separate components, it would be undesirable from an aesthetic standpoint for the camera body 1, and it would also be undesirable in terms of making it cumbersome for the user to put it around their neck. Therefore, it is desirable that the band portion 82 and the electrical cable 84 be integrated, but the configuration is not limited to the one shown in Figure 2B.

[0709] Figure 45 shows the band portion 82 and the connection surface 83, which is a cross-section of the electrical cable 84 that is integrally formed therewith.

[0710] Figures 45(a) to (c) show the case where the electrical cable 84 is made of a flexible printed circuit board (FPC), and Figures 45(d) to (g) show the case where the electrical cable 84 is made of a thin wire cable.

[0711] In Figures 45(a) and (d), the electrical cable 84 is embedded inside the band portion 82 as viewed from the connection surface 83. In this case, it is desirable that the material of the band portion 82 be an injection-molded elastic material such as silicone rubber, elastomer, rubber, or plastic. One method of manufacturing a component in which the band portion 82 and the electrical cable 84 are integrated is to insert the electrical cable 84 during the injection molding of the band portion 82. Alternatively, a manufacturing method may be adopted in which the band portion 82 is composed of two parts, the electrical cable 84 is sandwiched between them, and the two parts of the band portion 82 are integrated into a single component by adhesive or heat welding. However, as shown in Figures 45(a) and (d), it is sufficient that a component in which the band portion 82 and the electrical cable 84 are integrated can be manufactured, and the manufacturing method is not limited to the two methods described above.

[0712] Figures 45(b), (c), and (e) show that the electrical cable 84 is in contact with the outside of the band portion 82, as viewed from the connection surface 83.

[0713] In Figure 45(b), the band portion 82, as viewed from the connection surface 83, does not have any special shape for integrating with the electrical cable 84; the electrical cable 84 is simply attached to the surface of the band portion 82. Because it is constructed using adhesive bonding, it can be manufactured at a low cost. However, if the electrical cable 84 (in this case, the FPC) is on the external side, the exposure of the FPC may be undesirable from an aesthetic standpoint. In this case, treatment such as painting or covering the exterior with a film is required to eliminate the exposure of the FPC. Also, if the electrical cable is on the side that comes into contact with the user's neck, it may result in poor wearing comfort. In this case, treatment such as painting or covering the exterior with a film is also required to improve wearing comfort.

[0714] In Figures 45(c) and (e), the band portion 82, as viewed from the connection surface 83, is provided with a recessed shape 83a to integrate with the electrical cable 84, and the electrical cable 84 is positioned inside this recessed shape 83a. In this case, if the recessed shape 83a is provided on the side that contacts the user's neck, the aesthetic appearance of the camera body 1 is ensured, and if the recessed shape 83a is provided so that the electrical cable 84 does not directly contact the user's neck, a good fit for the user can be maintained without any special processing. Furthermore, if the band portion 82 is properly designed during manufacturing, there is no additional cost for adding the recessed shape 83a, making it cost-effective.

[0715] In Figures 45(f) and (g), as with Figures 45(a) and (d) mentioned above, the electrical cable 84 is embedded inside the band portion 82 when viewed from the connection surface 83. Figure 45(f) shows the configuration when there is one electrical cable 84, and Figure 45(g) shows the configuration when there are multiple electrical cables 84. These configurations differ from Figures 45(a) and (d) in that they ensure a sufficient cross-sectional area of ​​the band portion 82 at the connection surface 83. The cross-sectional area of ​​the band portion 82 at the connection surface 83 affects the torsional rigidity and bending rigidity of the band portion 82, and these rigidities affect the stability of the shooting / detection unit 10 so that it remains stable in a fixed position on the body when the user attaches the camera body 1. In other words, the larger the cross-sectional area of ​​the band portion 82 at the connection surface 83, and the stronger the torsional rigidity and bending rigidity, the better the stability of the shooting / detection unit 10. Furthermore, if the protruding side of the electrical cable 84 is directed towards the user's neck, the contact area between the user and the band portion 82 will decrease, resulting in a poor fit. Therefore, it is preferable that the protruding side of the electrical cable 84 be positioned towards the external side. Accordingly, the shapes shown in Figures 45(f) and (g) are less aesthetically pleasing than the other shapes shown in Figure 45 because the protruding shape of the electrical cable 84 on the band portion 82 is visible from the external side. However, these shapes are suitable from the viewpoint of ensuring the rigidity of the band portion 82.

[0716] Based on the above, the shapes shown in Figures 45(c) and (e) are considered the most suitable in terms of balancing aesthetics and wearing comfort. However, other shapes shown in Figure 45 can be used if cost or rigidity is a priority.

[0717] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence.

[0718] (Example 9) In Example 9, a modified example of the camera system including the camera body 1 will be described using Figures 46A and 46B.

[0719] This embodiment will be described as a derivative of Embodiment 1. Therefore, for the camera system configuration of Embodiment 9, the same reference numerals will be used for components identical to those of the camera system in Embodiment 1, and redundant explanations will be omitted. For components that differ, details will be added as needed.

[0720] In Example 1, a standard smartphone was used as the display device 800, but there are multiple smartphone manufacturers on the market, and their processing capabilities vary widely. For example, in Example 1, when transferring video in the recording direction extracted from an ultra-wide-angle image to the display device 800, information necessary for optical correction processing and image stabilization processing was added to the video. Based on this information, the display device While the 800 performs distortion correction and image stabilization, the processing power of a smartphone used as the 800 display device may be insufficient to perform these corrections.

[0721] Therefore, the camera system of this embodiment includes a camera body 1' as an imaging device and a display device 9800 with lower processing power than the display device 800.

[0722] In camera body 1', once the initial video recording process (steps S100 to S600 in Figure 7A) is completed, the transfer process to the display device 9800 is not performed, and the processes in steps S800 and S900 are executed. Subsequently, camera body 1' transfers the video, for which the processes in steps S800 and S900 have been completed, to the display device 9800.

[0723] On the other hand, the display device 9800 records the video from the camera body 1' directly without performing the processing in steps S800 and S900.

[0724] The camera system of this embodiment will be described in detail below.

[0725] Figure 46A is a block diagram showing the hardware configuration of the display device 9800 connected to the camera body 1', which serves as the imaging device in this embodiment.

[0726] In Figure 46A, components identical to those in the hardware configuration of the display device 800 according to Embodiment 1 shown in Figure 6 are denoted by the same reference numerals, and redundant explanations are omitted.

[0727] The display device 9800 differs from the display device 800 in that it has a display device control unit 9801 instead of the display device control unit 801, and does not have a face sensor 806.

[0728] The display control unit 9801 is configured with a CPU that has lower processing power than the CPU that constitutes the display control unit 801 (Figure 6). Furthermore, the capabilities of the built-in non-volatile memory 812 and primary memory 813 may be lower than those in Example 1.

[0729] Figure 46B is a functional block diagram of the camera body 1'.

[0730] In Figure 46B, components identical to those in the camera body 1 according to Embodiment 1 shown in Figure 4 are denoted by the same reference numerals, and redundant explanations are omitted.

[0731] The functional blocks shown in Figure 46B differ from those in Figure 4 in that they include an optical / vibration correction unit 9080 that performs optical correction processing and vibration damping processing, and that each functional block is executed under the control of the overall control CPU 9101 rather than the overall control CPU 101. Another difference is that one of the predetermined communication partners of the transmission unit 70 is the display device 9800 rather than the display device 800.

[0732] In other words, in this embodiment, the optical and vibration-damping correction unit 9080 of the overall control unit CPU 9101 performs optical distortion correction and vibration-damping correction using optical correction values ​​and gyro data. Therefore, compared to the video file 1000 that the transmission unit 70 transfers to the display device 800 in Embodiment 1, the video file after optical distortion correction and vibration-damping correction that the transmission unit 70 transfers to the display device 9800 in this embodiment has a smaller data size.

[0733] Furthermore, since the display device 9800 does not perform the processing in steps S800 and S900, its processing power can be kept lower compared to the display device 800. In addition, the image captured by the camera body 1' can be viewed on a simple display device 900 (viewing unit) consisting of a smartwatch or the like.

[0734] The above describes a method for recording an image of the area observed by the user using the camera body 1. The area extracted by the camera body 1 will be referred to as the extracted area below. The extracted area may be outside the area the user wanted to record in some parts of the video, such as when the user turns their face away. Examples 10 and 11 are examples of a method for transferring the video captured by the camera body 1 to the display device 800 and correcting the extracted area.

[0735] (Example 10) Figure 50 is an example diagram illustrating a portion of a video clip. The display device 800 receives video footage shot by the user in video mode on the camera body 1 and stores it in the large-capacity non-volatile memory 814. Figure 50 shows an example in which the display device 800 displays the cropped area 5000 determined by the camera body 1 for 5 frames of the video.

[0736] Figure 50(a) shows the subject 131 that the user wants to keep as a video, positioned in the center of the cropping area 5000. Figure 50(b) shows the subject 131 moving towards the edge of the cropping area 5000. Figure 50(c) shows the subject 131 going out of frame of the cropping area 5000. Figure 50(d) shows the subject 131 returning to the center of the cropping area 5000. Figure 50(e) shows the subject 131 returning to the center of the cropping area 5000.

[0737] When imaging a child as the subject 131, the user's face direction may deviate from the subject 131 if they turn their face towards a passing car or towards a sound. In this case, as shown in Figure 50(c), the subject 131 may be cut off from the cropped area.

[0738] In Example 10, in order to allow the display device 800 to correct the cropped area of ​​the video captured by the camera body 1 after capture, the camera body 1 has a full-area video mode as an imaging mode that records the captured video without applying cropping processing.

[0739] The imaging modes of the camera body 1 will be explained with reference to Figure 51. Figure 51 is a view of the imaging / detection unit 10 according to Embodiment 10, seen from the back side (the side that comes into contact with the user). The configuration other than the imaging mode switch 12A is the same as in Figure 2C, so the explanation will be omitted. The imaging mode switch 12A is a slide lever type switch that allows selection of the imaging modes "Photo," "Normal," and "Pri," the same as in Figure 2C, as well as "Whole," which corresponds to the full-range video mode.

[0740] Referring to Figures 52A to 52D, the process of recording the entire video from the video captured by the camera body 1 without cropping a region and transferring it to the display device 800 will be described. The processes in Figures 52A to 52D correspond to the processes in Figures 7A, 7B, 7E, and 14 of Embodiment 1, respectively, and similar processes are denoted by the same reference numerals, and detailed explanations are omitted.

[0741] In Example 10, the camera body 1 captures images in full-area video mode with the imaging mode switch 12A set to "Whole". The camera body 1 records the video of the entire area without cropping and transfers it to the display device 800.

[0742] In step S8100 in Figure 52A, the overall control CPU 101 executes the preparation operation process corresponding to step S100 in Figure 7A. Figure 52B is a flowchart of the subroutine for the preparation operation process in step S8100. In the preparation operation process in Figure 52B, in step S8102, if the mode selected by the imaging mode switch 12A is the full-range video mode, the process proceeds to step S103, similar to when the video mode is selected. .

[0743] In step S103, the overall control CPU 101 reads various settings for the full-range video mode from the built-in non-volatile memory 102 and saves them to the primary memory 103. These settings for the full-range video mode include settings such as the vibration damping level. In step S104, the overall control CPU 101 starts the operation of the imaging driver 41 for the full-range video mode and finishes the preparation operation process.

[0744] In steps S200 to S400 of Figure 52A, the overall control CPU 101 performs the same processing as in steps S200 to S400 of Figure 7A. In step S400, the imaging unit 40 takes an image and generates imaging data.

[0745] In step S8500, the image cropping and development processing unit 50 performs development processing on the entire area of ​​the image data generated in step S400. Figure 52C is a flowchart of the subroutine for the full-area development processing in step S8500. The full-area development processing in Figure 52C is the same as the flow shown in Figure 7E in Example 1, but with the area cropping processing (step S502) removed.

[0746] In step S501, similar to step S501 in Figure 7E, the image cropping and development processing unit 50 acquires RAW data for the entire area. In steps S8503 to S8508, the image cropping and development processing unit 50 performs the same processing as in steps S503 to S508 in Figure 7E for the entire area.

[0747] In step S8600 of Figure 52A, the overall control CPU 101 saves the entire video area developed in step S8500 to the primary recording unit 60. Figure 52D is a flowchart of the subroutine for the primary recording process in step S8600. The differences between the primary recording process in Figure 52D and the primary recording process in Embodiment 1 shown in Figure 14 will be explained. The primary recording process shown in Figure 52D corresponds to steps S601 to S606 in Figure 14 and is executed for each frame of the video developed in step S8500. In Figure 52D, the frame to be processed is called the current frame.

[0748] In step S601 of Figure 52D, the overall control CPU 101, similar to step S601 of Figure 14, attaches (adds) information about the cropping position (position of the cropping region) of the current frame to the correction data, which is metadata for the current frame. The attached cropping position information is, for example, the coordinates Xi,Yi of the video recording frame 127i recorded in step S305 of Figure 7D.

[0749] In step S8601, the overall control CPU 101 also attaches (adds) the cropping size (size of the cropping area) of the current frame's image to the correction data. The attached information on the cropping position is, for example, the width WXi and height WYi of the video recording frame 127i recorded in step S305 in Figure 7D.

[0750] The processing in steps S602 to S605 is the same as the processing indicated by the same reference numerals in Figure 14. In step S8606, the correction data (frame metadata) for the current frame includes not only the position of the cropping region but also the size of the cropping region. The overall control CPU 101 generates a video file for transfer to the display device 800 by executing the processing shown in Figure 52D for each frame of the video developed in step S8500. By including the position and size of the cropping region of each frame in the video file, the display device 800 can perform cropping processing on the video file received from the camera body 1.

[0751] Referring to Figure 53, the data structure (format) of the video file will be explained. The video file 2000 (video data) shown in Figure 53 includes a header 1001 and a frame 2002, similar to the video file 1000 described in Figure 15. Frame 2002 contains frame datasets, each containing an image of one frame that makes up the video and its corresponding frame metadata, for the total number of frames in the video. The image of each frame in video file 2000 is a full-area image.

[0752] The frame metadata for each frame includes information about the crop size, which is attached as correction data in step S8601. Furthermore, the frame metadata of the video file 2000 includes three additional parameters, in addition to those described in Figure 15: crop position correction amount, crop size correction amount, and user change operation flag. These three additional parameters are used in the process of correcting the cropped area of ​​the video on the display device 800.

[0753] The user modification flag indicates whether the cropped area has been corrected by user operation. The cropped position correction amount is the amount of correction to the position of the cropped area corrected by user operation. The cropped size correction amount is the amount of correction to the size of the cropped area corrected by user operation.

[0754] In the primary recording process of step S8600, the cropping position correction amount and cropping size correction amount are set to 0 for all frames. The user change operation flag is set to ON for the first and last frames of the video, and to OFF for all other frames. The user change operation flag is set to ON for the first and last frames of the video because they are used to obtain (calculate) the correction amount for the cropping area of ​​frames in which the user did not perform any cropping area correction operations.

[0755] Note that the frames in which the user change operation flag is set to ON are not limited to the first and last frames of the video. If the video contains multiple scenes, the user change operation flag may be set to ON in the last frame before the scene changes and in the first frame after the scene changes. The display device control unit 801 can detect scene changes by various methods, such as when the subject changes due to video analysis, or when the difference in pixel values ​​between frames exceeds a threshold.

[0756] The display device control unit 801 can correct the cropping area based on the first and last frames of the scene containing the frame to be corrected, even if the subject changes before and after the scene changes. As a result, the display device control unit 801 can generate cropped videos in which the subject is appropriately cropped according to the scene.

[0757] The processing in steps S700 to S1000 in Figure 52A is the same as the processing in steps S700 to S1000 in Figure 7A. The video file 2000 (video data) that was initially recorded in step S8600 is transferred to the display device 800 in step S700. The display device control unit 801 of the display device 800 performs optical correction processing and vibration damping processing on the video file 2000 received from the camera body 1 in steps S800 and S900. The display device control unit 801 stores the video file 2000 in the large-capacity non-volatile memory 814. Through the processing shown in Figure 52A, the video data of the entire area captured by the camera body 1 is stored in the large-capacity non-volatile memory 814 of the display device 800.

[0758] Referring to Figure 54, the correction of the cropped area in the display device 800 will be explained. The full-area video display unit 881 is the area that displays the full-area video stored in the display device 800. The cropped area frame 882 indicates the area that has been determined to be the cropped area in the full-area video.

[0759] The position and size information of the cropping area frame 882 is determined in step S300 of Figure 52A and stored as frame metadata in the video data. The display device control unit 801 reads the information of the cropping area of ​​the frame from the video file 2000 for the frame image being displayed and controls the display to superimpose the cropping area frame 882, which indicates the cropping area, onto the frame image.

[0760] The user can correct the cropped area by manipulating the cropped area frame 882. Correcting the cropped area includes changing at least one of the position and size of the cropped area.

[0761] The user can move the cropping area by dragging the cropping area frame 882 on the touch panel (display unit 803 of the display device 800). The operation to move the cropping area is not limited to dragging the cropping area frame 882, but may also be an operation to tap the destination of the cropping area frame 882 or the subject 131 to be captured.

[0762] Furthermore, the user can change the size of the cropping area by pinching in or out on the cropping area frame 882 on the touch panel.

[0763] The playback speed selection unit 883 is a selection member (item) for selecting the playback speed when playing a video. The play / pause button 884 is an operation member (item) for playing or pausing the video displayed on the full-area video display unit 881.

[0764] The play / pause button 884 functions as a play button when no video is playing on the full-area video display unit 881. When the user selects the play button, the video displayed on the full-area video display unit 881 is played at the playback speed specified by the playback speed selection unit 883. The play / pause button 884 functions as a pause button when a video is playing on the full-area video display unit 881. When the user selects the pause button, playback of the video playing on the full-area video display unit 881 is paused.

[0765] The seek bar 885 is an operating element (item) that allows the user to display any frame of the video displayed on the full-area video display unit 881 by sliding the slider 888. The full-area video display unit 881 displays the image of the frame selected by the user using the slider 888, along with the corresponding cropping area frame 882.

[0766] The Confirm button 886 is an operating element (item) used to confirm the user's correction of the cropping area. Selecting (tapping) the Confirm button 886 confirms the user's movement of the cropping area frame 882. When the Confirm button 886 is selected and the position of the cropping area is confirmed by moving the cropping area frame 882, the position of the cropping area frame 882 in the frames before and after the currently displayed frame is also corrected. Note that if the user has not moved the cropping area frame 882 from its initial or confirmed state, the Confirm button 886 will be grayed out to indicate that the operation is invalid.

[0767] The generate button 887 is an operation component (item) for generating a cropped video by extracting each cropped area from the image of each frame of the full-range video. Note that the generate button 887 will be grayed out to indicate that the operation is invalid if the video is playing or if the correction of the cropped area frame 882 has not been finalized.

[0768] Refer to Figures 55 to 58, and based on the user's correction operation of the cropped area, the display device This section outlines how the 800 generates extracted video clips. Figures 55(a) to 55(e) show the full-area video of the same scene as the extracted video clips shown in Figures 50(a) to 50(e), but captured without performing the extraction process.

[0769] The user operates the seek bar 885 from the state before playback starts, as shown in Figure 55(a), to find a frame in which the subject 131 is shifted from the cropping area frame 882, as shown in Figure 55(c).

[0770] Figure 56 shows an example of a user dragging the cropping area frame 882. As shown in Figure 55(c), the user can move the cropping area frame 882 by finding a frame where the subject 131 is misaligned with the cropping area frame 882 and dragging it on the touch panel. In the example in Figure 56, the cropping area frame 882 is moved from its original position enclosed by the dashed line so that the subject 131 fits within the cropping area frame 882. Once the movement of the cropping area frame 882 is complete, the user selects the confirm button 886.

[0771] Figure 57 illustrates the correction of the cropped area of ​​an unoperated frame. An unoperated frame is a frame in which the user has not yet performed an operation to correct the cropped area. A frame in which the user has performed an operation to correct the cropped area is called a corrected frame. If, for the frame shown in Figure 55(c), the user moves the cropped area and selects the confirmation button 886 as explained in Figure 56, the display device control unit 801 also corrects the cropped areas of the unoperated frames before and after the frame shown in Figure 55(c).

[0772] Figures 57(a) and 57(b) correspond to Figures 55(b) and 55(d), which are the unoperated frames before and after the frame in Figure 55(c). In the frame shown in Figure 57(a), the cropping area frame 882 is moved from its original position enclosed by the dashed line in the direction in which the user moved the cropping area frame 882 in Figure 56. Similarly, in the frame shown in Figure 57(b), the cropping area frame 882 is moved from its original position enclosed by the dashed line in the direction in which the user moved the cropping area frame 882 in Figure 56.

[0773] In this way, the display device control unit 801 automatically corrects the cropped area of ​​the unoperated frame to match the correction amount (movement amount or size change amount) of the cropped area in the correction frame, without requiring the user to directly correct it. By correcting the cropped area of ​​the unoperated frame to match the correction of the cropped area in the correction frame, the display device control unit 801 can generate cropped video so that the cropped area moves smoothly.

[0774] The user searches for a frame where the subject 131 and the cropping area frame 882 are misaligned, corrects the cropping area frame 882, and confirms the result. The cropping area of ​​an unoperated frame is corrected in accordance with the correction of the cropping area in the corrected frame when the confirm button 886 is selected. Note that the cropping area of ​​an unoperated frame is not limited to when the confirm button 886 is selected; it may also be corrected when the user moves the cropping area in the currently displayed frame and then switches to another frame.

[0775] If the user determines that there are no more frames that need correction in the cropping area frame 882, they can select the generate button 887 to generate a cropped video containing the corrected cropped areas of each frame from the entire video.

[0776] Figures 58(a) to 58(e) illustrate the extracted video generated by the display device 800. In the full-area video shown in Figures 55(a) to 55(e), if the user corrects the extracted area as shown in Figure 56, the display device control unit 801 will perform the following actions as shown in Figures 58(a) to 58(e). A cropped video can be generated. By correcting the cropping regions in Figures 55(b) and 55(d) to match the user's corrections to the cropping region in Figure 55(c), the display device control unit 801 can generate a cropped video so that the subject 131 is included in the cropping region in each frame. The generated cropped video is stored in the large-capacity non-volatile memory 814.

[0777] Referring to Figures 59A, 59B, and 59C, the flow of video editing processing by the display device 800 according to Embodiment 10 will be explained. Figure 59A is a flowchart of the video editing process in which the cropped area is corrected in editing mode.

[0778] In step S9001, the display device control unit 801 acquires the first frame image from the video file 2000 as described in Figure 53. The image acquired from the video file 2000 is an image of the entire area. In step S9002, the display device control unit 801 displays the acquired image on the display unit 803 of the display device 800 (the full-area video display unit 881 in Figure 54).

[0779] In step S9003, the display device control unit 801 obtains the cropping position and cropping size of the first frame from the video file 2000. The cropping position and cropping size recorded in the video file 2000 are the default cropping position and cropping size determined based on the face direction and field of view setting values ​​relative to the camera body 1.

[0780] In step S9004, the display device control unit 801 displays a default cropping area frame 882 on the full-area video display unit 881 based on the cropping position and cropping size of the cropping area acquired in step S9003. The cropping area frame 882 is displayed as a rectangle surrounding the cropping area, for example, as shown in Figures 55(a) to 55(e).

[0781] In step S9005, the display device control unit 801 waits for user input. If it determines that user input has been received, the process proceeds to step S9006. In step S9006, the display device control unit 801 determines what kind of user input was received. In Figure 59A, there are four user inputs: playback operation using the play / pause button 884, operation of the slider 888 on the seek bar 885, drag operation of the cropping area frame 882, and cropping video generation operation using the generate button 887.

[0782] If, in step S9006, it is determined that the user operation is to play the video by selecting the play / pause button 884, the process proceeds to step S9007. In step S9007, the display control unit 801 changes the display of the play / pause button 884 to a display representing "pause". For example, the display control unit 801 changes the display of the play / pause button 884 from the display representing "play" in Figure 55(a) to the display representing "pause" in Figure 55(b).

[0783] In step S9008, the display device control unit 801 plays the video according to the playback speed selected by the playback speed selection unit 883. Similar to the first frame described in steps S9001 to S9004, the display device control unit 801 sequentially acquires the image, cropping position, and cropping size of each frame from the video file 2000, and displays the image and cropping area frame 882.

[0784] Furthermore, if information on the cropping position correction amount and cropping size correction amount is set in the frame metadata, the display device control unit 801 corrects the cropping position and cropping size using the cropping position correction amount and cropping size correction amount. The display device control unit 801 displays the cropping area frame 882 with the corrected cropping position and cropping size on the full area video display unit 8 Display on 81.

[0785] In step S9009, the display unit 801 determines whether the user has paused video playback by selecting the play / pause button 884, or whether playback has ended by displaying the final frame. If the display unit 801 determines that playback has paused or ended, it proceeds to step S9010. If the display unit 801 determines that pause has not been selected and playback has not ended, it returns to step S9008 and continues video playback.

[0786] In step S9010, the display control unit 801 changes the display on the play / pause button 884 to a display indicating "play". For example, the display control unit 801 changes the display on the play / pause button 884 from the display indicating "pause" as shown in Figure 55(d) to the display indicating "play" as shown in Figure 55(e). After changing the display on the play / pause button 884, the display control unit 801 returns to step S9005 and waits for user input.

[0787] If, in step S9006, it is determined that the user operation is to search for the frame to be corrected by operating the slider 888 on the seek bar 885 and display it on the full-area video display unit 881, the process proceeds to step S9011.

[0788] In step S9011, the display device control unit 801 obtains information from the video file 2000 regarding the frame image, cropping position, cropping size, cropping position correction amount, and cropping size correction amount corresponding to the position of the slider 888 of the seek bar 885. Using the information obtained from the video file 2000, the display device control unit 801 displays the cropping area frame 882, corrected by the frame image and correction amount corresponding to the position of the slide...

Claims

1. Acquisition means for acquiring video data in which information about the video's cropping region, which is set based on the face orientation of the user shooting the video, is added to the video. A display control means that displays a frame indicating the cropped area when the aforementioned video data is played back, An operating member that receives operations on the frame for correcting the cut-out area, Correction means for adding correction information indicating the amount of correction of the cropped area corrected by the above operation to the video data, A generation means that generates a cut video from the video based on the information of the cut region and the correction information. It has, The correction information includes at least one of the following: a cropping position correction amount, which is the amount of correction for the position of the cropped area corrected by user operation, and a cropping size correction amount, which is the amount of correction for the size of the cropped area corrected by user operation. The generation means uses the correction information for the correction frame, which is the frame that received the operation, to correct the cutout region of the unoperated frame, which is the frame that did not receive the operation, and generates the cutout video. A display device characterized by the following features.

2. The generation means interpolates the correction information for a first correction frame that precedes the unoperated frame among the correction frames and the correction information for a second correction frame that follows the unoperated frame among the correction frames to correct the cut-out region of the unoperated frame. The display device according to feature 1.

3. The first correction frame is the first frame of the video, or the first frame of a scene in the video that contains the unprocessed frame. The display device according to feature 2.

4. The second correction frame is the last frame of the video, or the last frame of the scene in the video that contains the unprocessed frame. The display device according to claim 2 or 3.

5. The generation means interpolates the correction information for a first predetermined number of frames prior to the unoperated frame and the correction information for a second predetermined number of frames after the unoperated frame to correct the cut-out region of the unoperated frame. The display device according to feature 1.

6. The generation means corrects the extracted region of the unoperated frame by at least one of the following methods: linear interpolation, cubic function interpolation using ease-in and ease-out, and multipoint interpolation. The display device according to any one of claims 2 to 5.

7. The correction of the cropped area includes changing at least one of the position and size of the cropped area. The display device according to any one of claims 1 to 6.

8. The correction information includes user modification operation flag information indicating whether or not the cut-out region has been corrected by the user operation. The display device according to any one of claims 1 to 7.

9. The operation described above is the operation of dragging the frame to the destination of the cropped area. The display device according to any one of claims 1 to 8.

10. The display device according to any one of claims 1 to 8, characterized in that the operation is an operation to tap the destination of the cropping area or the subject to be imaged.

11. The operation described above is the operation of pinching in or pinching out the frame. The display device according to any one of claims 1 to 8.

12. The operation described above is the operation of touching the destination of the frame or the subject being captured while the video is playing. The display device according to any one of claims 1 to 8.

13. The operation described above is the operation of pinching in or pinching out the frame while the video is playing. The display device according to any one of claims 1 to 8.

14. The aforementioned video is played at a speed less than 1x speed. The display device according to feature 12 or 13.

15. The aforementioned video data includes audio data collected by multiple microphones when the video was being filmed. The generation means changes the balance of the audio data collected by the multiple microphones in accordance with the correction of the cropped area. The display device according to any one of claims 1 to 14.

16. An acquisition step of acquiring video data in which information about the video's cropping region, which is set based on the face direction of the user shooting the video, is added to the video; A display control step that displays a frame indicating the cropped area when the aforementioned video data is played back, An operation step that accepts an operation on the frame for correcting the cropped area, A correction step of adding correction information indicating the amount of correction of the cropped area corrected by the above operation to the video data, A generation step of generating a cut video from the video based on the information of the cut region and the correction information. It has, The correction information includes at least one of the following: a cropping position correction amount, which is the amount of correction for the position of the cropped area corrected by user operation, and a cropping size correction amount, which is the amount of correction for the size of the cropped area corrected by user operation. A control method for a display device, characterized in that in the generation step, the correction information for the correction frame, which is the frame for which the operation has been received, is used to correct the cutout region of the unoperated frame, which is the frame for which the operation has not been received, and thereby generates the cutout video.

17. A program for causing a computer to function as one of the means of a display device according to any one of claims 1 to 15.

18. A computer-readable storage medium storing a program for causing a computer to function as one of the means of the display device described in any one of claims 1 to 15.

Images