Display device, control method of display device, and computer-readable storage medium

Abstract

The present application relates to a display device, a control method of a display device, and a computer-readable storage medium. The display device includes: an acquisition unit configured to acquire moving image data acquired by adding information of a cut-out region of a moving image set based on a face direction of a user who captures the moving image to the moving image; a display control unit configured to display a frame indicating the cut-out region when reproducing the moving image data; an acceptance unit configured to accept an operation for correcting the cut-out region with respect to the frame; a correction unit configured to add correction information of the cut-out region corrected by the operation to the moving image data; and a generation unit configured to generate a cut-out moving image from the moving image based on the information of the cut-out region and the correction information.

Description

Technical Field

[0001] This invention relates to a display device and a method for controlling the display device. Background Technology

[0002] Typically, when using a camera to record video, the camera operator needs to keep the camera pointed in the direction of the video, so his or her hands are occupied by the video operation and cannot be used to handle any other operations. Furthermore, the camera operator focuses his or her attention on the video and therefore cannot focus his or her attention on the experience of the place.

[0003] For example, considering the operation of a camera, there exists a situation where a parent, acting as the camera operator, cannot play with the child while filming the child, and cannot film the child if the parent attempts to play with the child.

[0004] Furthermore, when considering focusing attention on videography, there are situations where the camera operator is unable to cheer or remember the details of the game, or where videography cannot be conducted while the camera operator is focused on watching the sporting event. Similarly, there are situations where, when filming during a group tour, the camera operator cannot experience the same level of impression as other members, and videography is neglected when the experience is prioritized.

[0005] As a method for handling this situation, there is a method where a motion camera is fixed to the head using a head-mounting accessory and takes pictures in the direction of observation, while the camera operator takes pictures without his or her hands being occupied by the camera operation. Alternatively, there is a method where a wide-range celestial camera is used to capture images, focusing on the experience during the experience, cutting out the desired video portion from the captured celestial video, and editing it after the experience, thus enabling the retention of a video of the experience.

[0006] However, in the former method, it is necessary to perform the following steps: Figure 47A The annoying movement of the head-mounted accessory attached to the main body of the sports camera 901 shown is due to its attachment to the head. Additionally, as... Figure 47B As shown, when the camera operator uses the head-mounting accessory 902 to equip the action camera 901 on their head, their appearance deteriorates, and the shape of their hair is disrupted. Furthermore, the camera operator is concerned about the weight of the head-mounting accessory 902 and the action camera 901 mounted on their head, or about the potential deterioration in appearance for third parties. For this reason, the following problems exist: Figure 47B In the state shown, the camera operator cannot focus on the experience, and the camera operator is in a state of... Figure 47BThe object exhibited showed resistance, making it difficult to take photographs.

[0007] On the other hand, the latter method requires a series of operations such as image transformation and specifying the cutout position. For example, it is known that... Figure 48 The image shown is of a celestial dome camera 903, including a lens 904 and a recording button 905. Lens 904 is one of a pair of fisheye lenses disposed on two sides of the housing of the celestial dome camera 903 for capturing images of a semi-celestial sphere, and the celestial dome camera 903 uses this pair of fisheye lenses to perform celestial dome photography. In other words, celestial dome photography is performed by combining the projected images from the pair of fisheye lenses.

[0008] Figures 49A to 49C This diagram illustrates an example of the conversion operation of video captured by the omnidirectional camera 903.

[0009] Figure 49A This is an example of video acquired using celestial dome camera 903, including camera operator 906, child 907, and tree 908 as subjects. The video is obtained by combining the projected images of a pair of fisheye lenses using a hemispherical optical system, resulting in significant distortion of camera operator 906. Furthermore, since the body part of child 907, the subject to be filmed by camera operator 906, is located near the hemispherical optical system, this body part is magnified and significantly distorted in the horizontal direction. On the other hand, tree 908, being the subject located in front of lens 904, is filmed with minimal distortion.

[0010] In order to Figure 49A The video shown is created to represent the visual field as seen by an average person. This requires cutting out a portion of the video, performing a planar transformation on it, and then displaying the resulting video.

[0011] Figure 49B Through from Figure 49A The video shown is the result of cutting out the video from the front of lens 904. Figure 49B In the video shown, tree 908 is shown in the center of the field of view as seen by an average person. However, the child 907, who was to be filmed by camera operator 906, is not included. Figure 49B Therefore, the cutout position needs to be changed.

[0012] Here, more specifically, it is necessary to Figure 49A The tree 908 seen in the image has its cutout position changed to the left and downwards at a 30° angle. The video displayed after this cutout operation, via planar transformation, is... Figure 49C In this way, in order to... Figure 49A The video shown is captured by the camera operator as they intended to film. Figure 49C The video shown requires operations to cut out the desired portions and perform planar transformations. For this reason, although the camera operator can focus on the experience during filming (during recording), there are situations where the amount of subsequent work becomes enormous.

[0013] Japanese Patent Application Publication No. 2017-212550 discloses a technique for correcting the cutout position of an image, which allows a user to change the deviation correction intensity when reproducing the image and determine the cutout size and cutout position of the image based on the deviation correction intensity.

[0014] Although the cutout area can be indirectly corrected by changing the deviation correction intensity, it is difficult for users to correct the cutout area to the expected position unless the cutout area is directly changed. Summary of the Invention

[0015] The purpose of this invention is to provide a technique for generating cut-out motion images at predetermined locations using simple and easy operations.

[0016] A display device according to the present invention includes: an acquisition unit configured to acquire motion image data, the motion image data being acquired by adding information of a cutout region of the motion image set based on the facial orientation of a user capturing the motion image to the motion image; a display control unit configured to display a frame representing the cutout region when reproducing the motion image data; a receiving unit configured to receive an operation for correcting the cutout region for the frame; a correction unit configured to add correction information of the cutout region corrected by the operation to the motion image data; and a generation unit configured to generate a cutout motion image from the motion image based on the information of the cutout region and the correction information.

[0017] Further features of the invention will become apparent from the following description of typical embodiments with reference to the accompanying drawings. Attached Figure Description

[0018] Figure 1A This is an external view of the camera body, including the imaging / detection unit, which is the imaging device according to Embodiment 1.

[0019] Figure 1B It is an image showing the appearance of the camera body worn by the user.

[0020] Figure 1C From Figure 1A A diagram of the battery unit of the camera body as seen from the rear.

[0021] Figure 1DThis is an external view of the display device, which is configured separately from the camera body, as a portable device according to Embodiment 1.

[0022] Figure 2A This is a schematic diagram of the camera / detection unit as seen from the front.

[0023] Figure 2B This is a diagram showing the shape of the strap portion of the connecting unit of the camera body.

[0024] Figure 2C This is a diagram of the camera / detection unit as seen from the rear side.

[0025] Figure 2D This is a view of the camera / detection unit from above.

[0026] Figure 2E This is a diagram showing the structure of the infrared detection processing device located below the face direction detection window inside the camera / detection unit within the camera body.

[0027] Figure 2F This is a diagram showing the user wearing the camera body as seen from the user's left side.

[0028] Figures 3A to 3C This is a diagram showing the details of the battery cell.

[0029] Figure 4 This is a functional block diagram of the camera body according to Embodiment 1.

[0030] Figure 5 This is a block diagram showing the hardware structure of the camera body.

[0031] Figure 6 This is a block diagram showing the hardware structure of the display device.

[0032] Figure 7A This is a flowchart illustrating an overview of the video recording processing according to Embodiment 1 performed in the camera body and display device.

[0033] Figure 7B It is based on Example 1 Figure 7A The flowchart shows the subroutine for the preparation operation process in step S100.

[0034] Figure 7C It is based on Example 1 Figure 7A The flowchart of the face orientation detection processing subroutine in step S200 is shown.

[0035] Figure 7D It is based on Example 1 Figure 7A The flowchart shows the subroutine for determining the recording direction / range in step S300.

[0036] Figure 7E It is based on Example 1 Figure 7A The flowchart shows the subroutine of the recording range imaging process in step S500.

[0037] Figure 7F This shows the motion picture mode. Figure 7A The diagram shows the processing steps S200 to S500.

[0038] Figure 8A This is a diagram showing the user's video as seen from the face-oriented detection window.

[0039] Figure 8B This diagram illustrates a scenario where fluorescent lights in an indoor setting are shown as a background in a video of a user viewed from the face detection window.

[0040] Figure 8C This diagram illustrates video acquired under the following conditions: The infrared LEDs of the infrared detection processing device are not illuminated, and the sensor of the infrared detection processing device is used through the face direction detection window to... Figure 8B The image shows the user and the fluorescent light as the background.

[0041] Figure 8D This diagram illustrates video acquired under the following conditions: with the infrared LED illuminated, the infrared detection processing device's sensor uses the face orientation detection window to... Figure 8B The image shows the user and the fluorescent light as the background.

[0042] Figure 8E It shows the basis Figure 8C and Figure 8D The image shown is a graph of the differential video calculated from the video.

[0043] Figure 8F The diagram illustrates the following situation: Figure 8E The intensity of the differential video shown is scaled to the light intensity of the reflected beam of infrared light projected onto the user's face and neck.

[0044] Figure 8G It is shown in the following image: Figure 8F The top layer shows markers representing different parts of the user's body, as well as double circles and black circles representing the neck and chin positions, respectively.

[0045] Figure 8H This is to show the use of [something] when the user's face is turned to the right. Figure 8E The method is similar to the method used to calculate the differential video graph.

[0046] Figure 8I It is shown in the following image: Figure 8H The double circles and black circles are superimposed to indicate the position of the neck and the position of the chin tip.

[0047] Figure 8J This is a diagram showing a video of the user as seen from the face orientation detection window when the user turns his or her face to a side 33° higher than the horizontal side.

[0048] Figure 8K The image below shows the following: When the user has turned his or her face to a side that is 33° higher than the horizontal side, when using... Figure 8E The method is similar to the one used to calculate the differential video, with double circles and black circles superimposed to indicate the neck position and the chin position, respectively.

[0049] Figures 9A to 9E This is a timing diagram showing the lighting timing of the infrared LED.

[0050] Figures 10A to 10D It is a diagram showing the vertical movement of the user's face.

[0051] Figure 11A This is a diagram showing the target field of view in an ultra-wide-angle video captured by the camera unit of the main camera body when the user is facing the front side.

[0052] Figure 11B This shows a cut-out from an ultra-wide-angle video. Figure 11A A diagram of the target field of view in the video.

[0053] Figure 11C This is a diagram showing the target field of view in an ultra-wide-angle video captured while the user is observing subject A.

[0054] Figure 11D This demonstrates how to correct the cut-out from an ultra-wide-angle video. Figure 11C The image shows the distortion and sway of the target field of view video obtained from the video.

[0055] Figure 11E It shows that the user is using a ratio Figure 11C The image shows the target field of view in the ultra-wide-angle video obtained when observing subject A with a small viewing angle setting.

[0056] Figure 11F This demonstrates how to correct the cut-out from an ultra-wide-angle video. Figure 11E The image shows the video obtained by distorting and swaying the target field of view.

[0057] Figure 12A This is a diagram illustrating an example of the target's field of view in an ultra-wide-angle video.

[0058] Figure 12BThis shows the viewpoint settings and... Figure 12A The target field of view shown has the same viewing angle settings and the same observation direction. Figure 12A The diagram shows examples of target fields of view with different viewing directions.

[0059] Figure 12C This shows the viewpoint settings and... Figure 12A The target field of view shown has the same viewing angle settings and the same observation direction. Figure 12A The figure shows another example of a target field of view with different viewing directions.

[0060] Figure 12D It shows the viewing direction and... in ultra-wide-angle video. Figure 12C The target fields of view shown have the same observation direction and the viewing angle setting value is less than 100%. Figure 12C The diagram shows an example of the target field of view with the shown view setting values.

[0061] Figure 12E It is shown in Figure 12A A diagram showing an example of assigning a preparation area near the target's field of view.

[0062] Figure 12F It is shown in Figure 12B The target field of view shown is assigned to the vicinity of the target. Figure 12E The diagram shows an example of a preparatory region with the same image stabilization level as the preparatory region.

[0063] Figure 12G It is shown in Figure 12D The target field of view shown is assigned to the vicinity of the target. Figure 12E The diagram shows an example of a preparatory region with the same image stabilization level as the preparatory region.

[0064] Figure 13 This is a diagram showing the menu screen of various settings for motion picture modes displayed on the display unit of the display device before the camera body takes pictures.

[0065] Figure 14 yes Figure 7A The flowchart of the main record processing subroutine for step S600 is shown.

[0066] Figure 15 This is a diagram illustrating the data structure of the video file generated using the master record processing.

[0067] Figure 16 Is Figure 7A The flowchart shows the subroutine of the display device's transmission processing performed in step S700.

[0068] Figure 17 yes Figure 7A The flowchart shows the subroutine of the optical correction process in step S800.

[0069] Figures 18A to 18F It is shown in Figure 17 The figure shows the distortion and aberration correction process in step S803.

[0070] Figure 19 yes Figure 7A The flowchart of the image stabilization subroutine shown in step S900.

[0071] Figure 20A and Figure 20B This is a diagram showing details of the calibrator used in the calibration process according to Example 2.

[0072] Figure 21 This is a flowchart of the calibration process according to Example 2, performed in the camera body and calibrator.

[0073] Figure 22A This demonstrates the calibration operation performed in the user's front-face orientation. Figure 21 The diagram shows the screen displayed in the display unit of the calibrator for step S3103.

[0074] Figure 22B It shows the user based on Figure 22A The indicated display shows a perspective view of the calibrator's appearance, indicating that the calibrator is held to the front.

[0075] Figure 22C It is shown in Figure 22B The diagram shows the entire ultra-wide-angle image perceived by the camera lens in the shown state.

[0076] Figure 22D This shows the correction Figure 22C This is a schematic diagram of the image obtained by aberrations in an ultra-wide-angle image.

[0077] Figure 22E This demonstrates that during calibration operations performed with the user's face facing forward, in Figure 21 This is a schematic diagram of the facial orientation image recorded by the facial orientation detection unit in step S3108.

[0078] Figure 22F It is shown in Figure 21 A schematic diagram of the judgment method used in step S3107 is shown.

[0079] Figure 23A This illustrates the process of performing a calibration operation from the user's right hand upwards. Figure 21The diagram shows the screen displayed in the display unit of the calibrator in step S3103.

[0080] Figure 23B It shows the user based on Figure 23A The indicated display shows a perspective view of the calibrator's appearance, with the indicators shown in the diagram indicating that it is held to the upper right side.

[0081] Figure 23C It is shown in Figure 23B The diagram shows the entire ultra-wide-angle image perceived by the camera lens in the shown state.

[0082] Figure 23D This shows the correction Figure 23C This is a schematic diagram of the image obtained by aberrations in an ultra-wide-angle image.

[0083] Figure 23E This demonstrates that during an upward calibration operation targeting the user's right hand, in Figure 21 This is a schematic diagram of the facial orientation image recorded by the facial orientation detection unit in step S3108.

[0084] Figures 24A to 24C This is a diagram showing a delayed cutout of an image according to Embodiment 3.

[0085] Figure 25A and Figure 25B This is a diagram showing the trajectory of the face stored in Example 3.

[0086] Figure 26 This is a flowchart of the treatment for preventing visually induced video motion sickness according to Example 3.

[0087] Figures 27A to 27F This is a conceptual diagram of the cutout range correction process based on Example 4.

[0088] Figure 28A and Figure 28B This is a flowchart of the cutout range correction process according to Example 4.

[0089] Figure 29A and Figure 29B This is a schematic diagram illustrating the relationship between the user's field of view and the target field of view when a subject located at close range is set as the object of observation in Embodiment 1.

[0090] Figure 30 This is an external view of the camera body of the imaging device according to Embodiment 5.

[0091] Figure 31 This is a block diagram showing the hardware structure of the camera body according to Embodiment 5.

[0092] Figure 32A and Figure 32B This is a schematic diagram illustrating the relationship between the user, the calibrator, and the target field of view during calibration, including parallax correction processing, according to Embodiment 5.

[0093] Figure 33A It is based on Example 5 as Figure 7A The flowchart shown is a part of the parallax correction mode processing in step S100, which is a preparation process.

[0094] Figure 33B It is based on Example 5 Figure 7A The flowchart shows the subroutine for determining the recording direction / range of S300.

[0095] Figures 34A to 34C It is shown Figure 33B A schematic diagram illustrating the relationship between the defocus image generated in step S5302 and the recording direction.

[0096] Figure 35 This is a flowchart for determining the processing direction based on the observation direction in Example 6.

[0097] Figure 36A This is a diagram illustrating the relationship between the user's viewing direction detection state for each frame and the captured video, according to Embodiment 6.

[0098] Figure 36B This is a diagram illustrating the relationship between the user's viewing direction detection state for each frame and the captured video in the subject loss mode according to Embodiment 6.

[0099] Figures 37A to 37C This is a diagram showing the relationship between the observation direction according to Example 7 and the facial region that can be used for detection in the facial direction.

[0100] Figure 38 It is a replacement Figure 35 The flowchart shows the observation direction determination process according to Embodiment 7, which is performed in step S6004.

[0101] Figure 39 This is a graph showing the relationship between facial orientation and facial orientation reliability according to Embodiment 7.

[0102] Figure 40 This is a conceptual diagram of the observation direction determination process when acquiring the facial orientation, based on Example 7.

[0103] Figures 41A to 41C This is an enlarged view showing the appearance of the camera / detection unit as seen from the side.

[0104] Figures 42A to 42C It is a side view showing the appearance of the camera body worn by the user.

[0105] Figures 43A to 43C This is an enlarged view showing the appearance of the camera / detection unit as seen from the side when the connection unit is set to be off-display.

[0106] Figures 44A to 44C This is a side view showing the appearance of the camera body when the connection unit is set to be off-screen.

[0107] Figures 45A to 45G This is a diagram showing the tape section and the connecting surface of the cable cut out as an integral part of the tape section.

[0108] Figure 46A This is a block diagram showing the hardware structure of a display device connected to the camera body of the imaging device according to Embodiment 9.

[0109] Figure 46B This is a functional block diagram of the camera body according to Embodiment 9.

[0110] Figure 47A and Figure 47B This is a diagram illustrating an example of the structure of a camera that is fixed to the head using a head-mounting accessory, which is a conventional technique.

[0111] Figure 48 This is a diagram illustrating an example of the structure of an all-sky dome camera, a conventional technology.

[0112] Figures 49A to 49C It shows that it is aimed at Figure 48 A diagram illustrating an example of the conversion operation of video captured by an all-sky dome camera.

[0113] Figures 50A to 50E This is a diagram showing the portion of the motion image that has been cut out.

[0114] Figure 51 This is a diagram showing the camera's shooting mode.

[0115] Figure 52A This is a flowchart illustrating an overview of the video recording process according to Embodiment 10.

[0116] Figure 52B This is a flowchart for the preparatory operation process.

[0117] Figure 52C This is a flowchart of the entire area imaging process.

[0118] Figure 52D This is a flowchart of the main record processing.

[0119] Figure 53 This is a diagram illustrating the data structure of a video file according to Embodiment 10.

[0120] Figure 54 This is a diagram showing the correction of the cutout area in the display device.

[0121] Figures 55A to 55E This is a diagram showing a motion image of the entire region as an example.

[0122] Figure 56 This is a diagram illustrating an example of a user dragging a cutout region box.

[0123] Figure 57A and Figure 57B This is a diagram showing the correction of the cut-out area of ​​the unoperated frame.

[0124] Figures 58A to 58E This is a diagram showing the cut-out motion image generated by the display device.

[0125] Figure 59A This is a flowchart of motion image editing processing according to Example 10.

[0126] Figure 59B This is a flowchart of the process for acquiring and processing correction information for frames that have not been operated on.

[0127] Figure 59C This is a flowchart of the process for generating moving images.

[0128] Figures 60A to 60E This is a diagram showing the correction of the cut-out area during reproduction.

[0129] Figure 61A This is a flowchart of motion image editing processing according to Example 11.

[0130] Figure 61B This is a flowchart of the process for acquiring and processing correction information for frames that have not been operated on. Detailed Implementation

[0131] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0132] Example 1

[0133] Figures 1A to 1D This diagram illustrates a camera system comprising a camera body 1, including an image / detection unit 10, which is an imaging device according to this embodiment, and a display device 800 disposed separately from the camera body 1. Although this embodiment shows the camera body 1 and display device 800 as separate configurations, they can be configured as a single unit. Furthermore, in the following text, the user who wears the camera body 1 around their neck will be referred to as the user.

[0134] Figure 1A This is an external view of the camera body 1.

[0135] refer to Figure 1A The camera body 1 includes an image / detection unit 10, a battery unit 90, and a connection unit 80 that connects the image / detection unit 10 and the battery unit 90 (power supply unit).

[0136] The camera / detection unit 10 includes a face orientation detection window 13, a start switch 14, a stop switch 15, a camera lens 16, an LED 17, and microphones 19L and 19R.

[0137] The face orientation detection window 13 enables the infrared LED lighting circuit 21 (built into the camera / detection unit 10) to illuminate. Figure 5 The infrared emitting unit in the middle emits infrared rays to detect the position of the user's face and the reflected beam passes through.

[0138] The power switch 14 is used to start recording video.

[0139] Stop switch 15 is used to stop the camera from recording.

[0140] The camera lens 16 guides the light beam used for imaging to the solid-state imaging element 42 inside the imaging / detection unit 10. Figure 5 ).

[0141] LED 17 is an LED indicating that the camera is on or a warning is being issued.

[0142] Microphones 19R and 19L are microphones that pick up ambient sound. Microphone 19L picks up sound from the user's left peripheral area (facing away from the camera). Figure 1A The microphone 19R picks up the user's right-side surroundings (facing the right side) of the sound. Figure 1A The sound (on the left side).

[0143] Figure 1B This is a diagram showing the appearance of the camera body 1 worn by the user.

[0144] With the camera body 1 worn such that the battery unit 90 is on the user's back and the camera / detection unit 10 is on the user's front, the camera body 1 is offset and held in the chest direction by a connecting unit 80 connected at both ends near the left and right ends of the camera / detection unit 10. Accordingly, the camera / detection unit 10 is located around the user's collarbone. At this time, the facial orientation detection window 13 is located below the user's jaw. Inside the facial orientation detection window 13, there is a [missing information - likely a device or structure]. Figure 2EThe infrared focusing lens 26 is shown. The optical axis of the camera lens 16 (imaging optical axis) and the optical axis of the infrared focusing lens 26 (detection optical axis) are oriented in different directions, and the user's viewing direction is detected based on the position of the facial portion using the face direction detection unit 20 (face direction detection unit) described below. Accordingly, imaging can be performed using the imaging unit 40 (imaging unit) described below, targeting the viewing direction.

[0145] The following will explain how to adjust the setting position and other aspects based on individual differences in body shape and clothing.

[0146] Furthermore, by arranging the camera / detection unit 10 on the front of the body and the battery unit 90 on the back, the weight is distributed, which reduces user fatigue and suppresses deviations caused by centrifugal force or other factors when the user moves.

[0147] Although this embodiment shows an example of the camera body 1 being worn such that the imaging / detection unit 10 is located around the user's collarbone, the configuration is not limited to this. Furthermore, as long as the camera body 1 can detect the user's viewing direction using the facial orientation detection unit 20 and can perform imaging in that viewing direction using the imaging unit 40, the camera body 1 can be worn on any part of the user's body other than the head.

[0148] Figure 1C From Figure 1A A schematic diagram of battery cell 90 seen from the rear.

[0149] refer to Figure 1C The battery unit 90 includes a charging cable insertion port 91, adjustment buttons 92L and 92R, and a spine avoidance notch 93.

[0150] The charging cable insertion port 91 is a charging cable insertion port not shown in the figure, and the internally arranged battery 94 is charged or the camera / detection unit 10 is fed power from an external power source through the charging cable.

[0151] Adjustment buttons 92L and 92R are used to adjust the lengths of the strap portions 82L and 82R of the connecting unit 80. Adjustment button 92L is used to adjust the left-facing strap portion 82L, and adjustment button 92R is used to adjust the right-facing strap portion 82R. In this embodiment, although the lengths of the strap portions 82L and 82R are adjusted independently by adjustment buttons 92L and 92R, the strap portions 82L and 82R can be adjusted simultaneously by a single button. Hereinafter, the strap portions 82L and 82R will be collectively referred to as strap portion 82.

[0152] The spine avoidance notch 93 is a notch designed to avoid contact between the spine and the user's spine, preventing the battery unit 90 from coming into contact with the spine. By avoiding the protrusions of the human spine, discomfort during wear is reduced, and horizontal movement of the main body is prevented during use.

[0153] Figure 1D This is an external view of the display device 800, which is a portable device according to Embodiment 1 and is configured separately from the camera body 1.

[0154] refer to Figure 1D The display device 800 includes a button A802, a display unit 803, a button B804, a built-in camera 805, a face sensor 806, an angular velocity sensor 807, and an acceleration sensor 808. Although in Figure 1D Not shown, but includes a wireless LAN that can be connected at high speed to the camera body 1.

[0155] Button A802 is a button that functions as the power button of the display device 800. It accepts the operation of power on / off by long press and accepts another processing timing indication by short press.

[0156] The display unit 803 can be used to inspect the video captured by the camera body 1 or to display the menu screen required for settings. In this embodiment, a transparent touch sensor is arranged on the surface of the display unit 803 and accepts operations based on touch on the displayed screen (e.g., the menu screen).

[0157] Button B804 is the button used as calibration button 854 in the calibration process described below.

[0158] The built-in camera 805 is a camera that can take pictures of people viewing the display device 800.

[0159] The face sensor 806 detects the shape of a person's face and the direction of their gaze in the observation display device 800. Although the specific structure of the face sensor 806 is not particularly limited, various sensors such as structured optical sensors, ToF sensors, and millimeter-wave radar can be used.

[0160] An angular velocity sensor 807 is arranged inside the display device 800, and is therefore represented by a dashed line in the perspective view scene. The display device 800 according to this embodiment also has the function of a calibrator, as will be described below; therefore, a three-dimensional gyroscope sensor capable of being used in three directions, including the X, Y, and Z directions, is installed in the display device 800.

[0161] Accelerometer 808 detects the posture of display device 800.

[0162] Since a general smartphone is used in the display device 800 according to this embodiment, the camera system according to the present invention can be implemented by associating the firmware in the smartphone with the firmware on the camera body 1 side. Here, the camera system according to the present invention can also be implemented by associating the firmware on the camera body 1 side with the application or OS of the smartphone used by the display device 800.

[0163] Figures 2A to 2F This is a diagram showing the camera / detection unit 10 in detail. In these and subsequent figures, the same reference numerals are assigned to the described parts to indicate the same functions, and the description of these functions in this specification will be omitted.

[0164] Figure 2A This is a view of the camera / detection unit 10 from the front.

[0165] In the connection unit 80, it is located on the right side of the user's body ( Figure 2A The right side connecting unit 80R (left side) and the left side of the user's body ( Figure 2A The left connecting unit 80L, which is composed of the right side of the camera / detection unit 10, is connected to the camera / detection unit 10. More specifically, the connecting unit 80 is divided into an angle-maintaining part 81 made of rigid material, which maintains the angle relative to the camera / detection unit 10, and a strap part 82. In other words, the right connecting unit 80R includes the angle-maintaining part 81R and the strap part 82R, and the left connecting unit 80L includes the angle-maintaining part 81L and the strap part 82L.

[0166] Figure 2B This is a diagram showing the shape of the belt portion 82 of the connecting unit 80. In this diagram, the angle maintaining portion 81 is shown in perspective to illustrate the shape of the belt portion 82.

[0167] The belt section 82 includes a connecting surface 83 and a cable 84.

[0168] The connecting surface 83 is the connecting surface between the angle maintaining part 81 and the belt part 82, and has a cross-sectional shape that is not perfectly circular (here, it is an elliptical shape). In the following text, the connecting surface 83 is located on the right side of the user's body (facing) when the camera body 1 is worn. Figure 2B (left side) and left side (facing) Figure 2B The connecting surfaces 83 arranged horizontally and symmetrically on the right side will be referred to as the right connecting surface 83R and the left connecting surface 83L, respectively. The right connecting surface 83R and the left connecting surface 83L happen to have the shape of the katakana character "ハ". In other words, in Figure 2BWhen viewed from the center, the distance between the right connecting surface 83R and the left connecting surface 83L decreases from the bottom upwards. Accordingly, when the user is wearing the camera body 1, the direction of the long axis of the connecting surface 83 of the connecting unit 80 is along the direction of the user's body, thus achieving the following effect: the strap 82 is comfortable when in contact with the user's body, and the camera / detection unit 10 does not move in the left / right or front / back directions.

[0169] Cable 84 (power supply unit) is a cable that is routed inside the belt section 82L and electrically connects the battery unit 90 and the camera / detection unit 10. Cable 84 connects the power supply of the battery unit 90 to the camera / detection unit 10 and transmits / receives electrical signals relative to the outside.

[0170] Figure 2C This is a schematic diagram of the camera / detection unit 10 as seen from the rear. Figure 2C It is from the side that makes contact with the user's body (that is, with...) Figure 2A The diagram shown is from the opposite side, therefore the positional relationship between the right connecting unit 80R and the left connecting unit 80L is... Figure 2A Their positional relationships are opposite.

[0171] The camera / detection unit 10 includes a power switch 11, a camera mode switch 12, and a chest connection pad 18 on its back side.

[0172] The power switch 11 is a power switch that toggles between turning the power on and off of the camera body 1. Although the power switch 11 according to this embodiment is a slider switch, the power switch 11 is not limited to this. For example, the power switch 11 may be a push switch, or it may be a switch integrally formed with a sliding cover of the camera lens 16 (not shown in the figure).

[0173] The camera mode switch 12 (changing unit) is a switch for changing the camera mode, and can change the mode related to camera recording. In this embodiment, in addition to still image mode and moving image mode, the camera mode switch 12 can also switch to the Pri setting (private setting) mode set by the display device 800, which will be described below. In this embodiment, the camera mode switch 12 is designed to be selected by sliding a lever. Figure 2C The switch 12 is a slider-type switch with one of the following options: "Photo," "Normal," and "Pri." The camera mode is switched from still image mode by sliding to "Photo," from moving image mode by sliding to "Normal," and from Pri setting mode by sliding to "Pri." The camera mode switch 12 is not limited to the form shown in this embodiment; it can be any switch capable of changing the camera mode. For example, the camera mode switch 12 could consist of three buttons: "Photo," "Normal," and "Pri."

[0174] When the camera / detection unit 10 applies force to the user's body, the chest connecting pad 18 (fixing unit) is the part corresponding to the user's body. For example... Figure 2A As shown, the camera / detection unit 10 is configured such that its entire horizontal length is longer than its entire vertical length when worn, and the chest connecting pad 18 is arranged near the left and right ends of the camera / detection unit 10. This arrangement suppresses horizontal rotation deviation of the camera body 1 during recording. Furthermore, the chest connecting pad 18 prevents the power switch 11 and the recording mode switch 12 from contacting the body. Additionally, the chest connecting pad 18 also prevents heat transfer to the user's body even when the temperature of the camera / detection unit 10 rises due to prolonged recording, and adjusts the angle of the camera / detection unit 10.

[0175] Figure 2D This is a view of the camera / detection unit 10 from above.

[0176] like Figure 2D As shown, the face orientation detection window 13 is arranged in the center of the surface of the camera / detection unit 10, and the chest connecting pad 18 protrudes from the camera / detection unit 10.

[0177] Figure 2E This is a diagram showing the structure of the infrared detection processing device 27 arranged inside the camera / detection unit 10 and below the face direction detection window 13.

[0178] The infrared detection and processing device 27 includes an infrared LED 22 and an infrared focusing lens 26.

[0179] Infrared LED 22 projects infrared light 23 towards the user. Figure 5 ).

[0180] Infrared focusing lens 26 is the reflected beam 25 emitted from the user when infrared light 23 is projected from infrared LED 22. Figure 5 The lens that images onto the sensor (not shown) in the figure of the infrared detection and processing device 27.

[0181] Figure 2F This is a diagram showing the user wearing the camera body 1 as seen from the user's left side.

[0182] An angle adjustment button 85L is a button arranged in the angle holding section 81L and is used to adjust the angle of the camera / detection unit 10. Although not shown in this figure, an angle adjustment button 85R is provided in a position symmetrical to the angle adjustment button 85L inside the angle holding section 81R, which is also arranged on the opposite side. Hereinafter, when collectively referred to as angle adjustment buttons 85R and 85L, these two will be referred to as angle adjustment buttons 85.

[0183] Although the angle adjustment button 85 exists in Figure 2A , Figure 2C and Figure 2D The position can also be seen in the image, but for the sake of simplicity, the angle adjustment button 85 has been omitted.

[0184] By pressing the angle adjustment button 85, the angle holding part 81 is oriented towards... Figure 2F By moving vertically, the user can change the angle between the camera / detection unit 10 and the angle maintaining unit 81. Additionally, the protrusion angle of the chest connecting pad 18 can be changed. Through the use of these two angle-changing components (angle adjustment button 85 and chest connecting pad 18), the camera / detection unit 10 can adjust the orientation of the camera lens 16 to a horizontal direction to accommodate individual differences in the position and shape of the user's chest.

[0185] Figures 3A to 3C This is a diagram showing details of battery cell 90.

[0186] Figure 3A The battery cell 90 is partially shown as a view from the back.

[0187] like Figure 3A As shown, in order to achieve weight balance in the battery unit 90, two batteries, including a left battery 94L and a right battery 94R (hereinafter also referred to as batteries 94), are symmetrically installed inside the battery unit 90. In this way, by symmetrically arranging the batteries 94 relative to the center of the battery unit 90, the left-right weight balance can be adjusted, and the position of the camera body 1 can be prevented from shifting. The battery unit 90 can be configured to house only one battery.

[0188] Figure 3B This is a view of battery cell 90 from above. Also in this view, battery 94 is shown in perspective.

[0189] like Figure 3B As shown, the relationship between the spine avoidance notch 93 and the battery 94 can be obtained. In this way, by arranging the battery 94 on both sides of the spine avoidance notch 93, a battery unit 90 with a relatively heavy weight can be installed without causing burden to the user.

[0190] Figure 3CThis is a diagram of battery cell 90 as seen from the back side. Figure 3C It is from the side that makes contact with the user's body (that is, with...) Figure 3A The image shown is from the opposite side of the view.

[0191] like Figure 3C As shown, the spine avoids the gap 93 and is arranged in the center along the user's spine.

[0192] Figure 4 This is a functional block diagram of the camera body 1. Details will be explained below; therefore, please refer to [reference needed here]. Figure 4 This will explain the entire processing flow that the camera body 1 needs to perform.

[0193] refer to Figure 4 The camera body 1 includes a face orientation detection unit, a recording direction / viewpoint determination unit 30, an imaging unit 40, an image cutting / display processing unit 50, a main recording unit 60, a transmission unit 70, and other control units 111. The overall control CPU 101 performs the overall control of the camera body 1. Figure 5 Such function blocks are executed under the control of ).

[0194] The facial orientation detection unit 20 (observation orientation detection unit) is a functional block executed by the infrared LED 22 and infrared detection processing device 27, etc., which estimates the observation orientation by detecting the facial orientation and transmits the observation orientation to the recording orientation / viewpoint determination unit 30.

[0195] The recording direction / viewpoint determination unit 30 (recording direction determination unit) performs various arithmetic operations based on the observation direction estimated by the face direction detection unit 20 to determine the position and range information when cutting out the video supplied from the camera unit 40, and transmits the information to the image cutting out / display processing unit 50.

[0196] The camera unit 40 converts the light beam sent from the subject into video and transmits the video to the image cutting / display processing unit 50.

[0197] The image cutting / display processing unit 50 (display unit) uses information supplied from the recording direction / viewpoint determination unit 30 to cut out the video supplied from the camera unit 40 and display it, thus only transmitting the video in the direction seen by the user to the main recording unit 60.

[0198] The main record unit 60 uses the main memory 103 ( Figure 5 The system consists of functional blocks such as ) which record video information and transmit the video information to the sending unit 70 at the required time intervals.

[0199] The transmitting unit 70 (video output unit) is wirelessly connected to the display device 800, which is a predetermined communication counterpart. Figure 1D It includes a calibrator 850 and a simple display device 900, and communicates with these three.

[0200] The display device 800 is a display device that can be connected to the transmitting unit 70 via a high-speed wireless LAN (hereinafter referred to as "high-speed wireless"). Here, in this embodiment, although radio communication corresponding to the IEEE 802.11ax (WiFi 6) standard is used for high-speed wireless, radio communication corresponding to another standard (e.g., WiFi 4 or WiFi 5) can also be used. Furthermore, the display device 800 can be a device specifically developed for the camera body 1, or it can be a general smartphone or tablet terminal, etc.

[0201] For the connection between the transmitting unit 70 and the display device 800, a low-power radio can be used, or the connection can be made using both high-speed wireless and low-power wireless, or by switching between the two. In this embodiment, high-speed wireless is used to transmit large amounts of data, such as video files of moving image videos (described below), and low-power wireless is used to transmit small amounts of data or data that takes longer to transmit. Here, although Bluetooth is used for low-power wireless in this embodiment, another near-field (short-range) wireless communication, such as Near Field Communication (NFC), can be used.

[0202] The calibrator 850 is a device for initial and individual setup of the camera body 1, and similar to the display device 800, it can be connected to the transmitting unit 70 via high-speed wireless connection. Details of the calibrator 850 will be described below. Additionally, the display device 800 may also have the functions of the calibrator 850.

[0203] For example, the simple display device 900 is a display device that can be connected to the transmitting unit 70 using only a low-power radio.

[0204] The simple display device 900 is a display device that, due to time constraints, cannot send moving image video to the transmitting unit 70, but can perform timed transmission for camera start / stop and image checks for composition confirmation. Furthermore, similar to the display device 800, the simple display device 900 can be a device specifically developed for the camera body 1, or a smartwatch, etc.

[0205] Figure 5 This is a block diagram showing the hardware structure of the camera body 1. (Referring to the reference...) Figures 1A to 1CThe structures and functions already described will use the same reference numerals, and their detailed descriptions will be omitted.

[0206] refer to Figure 5 The camera body 1 includes a general control CPU 101, a power switch 11, a camera mode switch 12, a face orientation detection window 13, a start switch 14, a stop switch 15, a camera lens 16, and an LED 17.

[0207] Additionally, the camera body 1 includes a face orientation detection unit 20. Figure 4 The infrared LED lighting circuit 21, infrared LED 22, infrared focusing lens 26, and infrared detection and processing device 27 are included.

[0208] Additionally, the camera body 1 includes an imaging unit 40 composed of an image driver 41, a solid-state imaging element 42, and an image signal processing circuit 43. Figure 4 ), and a transmitting unit 70 consisting of a low-power radio unit 71 and a high-speed radio unit 72. Figure 4 ).

[0209] In the camera body 1, although only one camera unit 40 is arranged in this embodiment, by arranging two or more camera units 40, 3D video can be captured, video with a wider field of view than that that can be obtained by one camera unit 40 can be captured, or video can be captured in multiple directions.

[0210] In addition, the camera body 1 includes various memory devices, such as a large-capacity non-volatile memory 51, a built-in non-volatile memory 102, and a main memory 103.

[0211] In addition, the camera body 1 includes a voice processing unit 104, a speaker 105, a vibrator 106, an angular velocity sensor 107, an acceleration sensor 108, and various switches 110.

[0212] The overall control CPU 101 connection has the above reference. Figure 2C The power switch 11, etc., is used to control the camera body 1. Figure 4 The recording direction / viewpoint determination unit 30, image cutting / display processing unit 50, and other control units 111 shown are composed of an overall control CPU 101.

[0213] Infrared LED lighting circuit 21 controls the above reference. Figure 2E The infrared LED 22 is turned on / off, and the projection of infrared rays 23 from the infrared LED 22 to the user is controlled.

[0214] The face direction detection window 13 is constructed using a visible light cutoff filter, which makes the visible light beam almost impossible to pass through, while allowing the infrared light 23, which is in the infrared region, and its reflected light beam 25 to pass through fully.

[0215] The infrared focusing lens 26 is a lens that focuses the reflected beam 25.

[0216] The infrared detection processing device 27 (infrared detection unit) includes a sensor that detects the reflected beam 25 focused by the infrared focusing lens 26. The sensor images the focused reflected beam 25 as video, converts the video into sensor data, and transmits the sensor data to the overall control CPU 101.

[0217] like Figure 1B As shown, when the user is wearing the camera body 1, the face orientation detection window 13 is located below the user's chin. For this reason, as... Figure 5 As shown, infrared rays 23 projected from the infrared LED lighting circuit 21 pass through the facial orientation detection window 13 and are emitted onto the infrared emitting surface 24 near the user's jaw. Furthermore, the infrared rays 23 reflected on the infrared emitting surface 24 become a reflected beam 25, which passes through the facial orientation detection window 13 and is focused by the infrared focusing lens 26 into the sensor present in the infrared detection processing device 27.

[0218] In Figure 1 to Figure 1C Various switches 110 are not shown in the diagram, and their details will be omitted. The various switches 110 are switches used to perform functions unrelated to this embodiment.

[0219] The camera driver 41 includes a timing generator, etc., which generates various timing signals, outputs the generated timing signals to camera-related units, and drives these units to perform camera operation.

[0220] Solid-state camera element 42 will transmit the image from the reference. Figure 1A The signal obtained by photoelectric conversion of the image of the subject projected by the camera lens 16 is output to the camera signal processing circuit 43.

[0221] The camera signal processing circuit 43 outputs the captured image data generated by clamping and AD conversion of the signal supplied from the solid-state camera element 42 to the overall control CPU 101.

[0222] The built-in non-volatile memory 102 uses flash memory or the like, and stores the startup program of the overall control CPU 1011 and setting values ​​for various program modes. In this embodiment, the effect level of changes in the field of view (viewing angle) and image stabilization control can be set, and such setting values ​​are also recorded.

[0223] The main memory 103 is constructed using RAM and the like, and temporarily stores the video data being processed, as well as the results of arithmetic operations performed by the overall control CPU 101.

[0224] The large-capacity non-volatile memory 51 records or reads main image data. In this embodiment, for the sake of simplicity, although the case of a semiconductor memory without an attachment / removal mechanism is described, the configuration is not limited to this. For example, the large-capacity non-volatile memory 51 may be constructed using an attachable / removable recording medium such as an SD card, and may also be used as a built-in non-volatile memory 102.

[0225] The low-power radio unit 71 uses a low-power radio to exchange data with the display device 800, the calibrator 850, and the simple display device 900.

[0226] The high-speed wireless unit 72 uses high-speed wireless to exchange data with the display device 800, the calibrator 850, and the simple display device 900.

[0227] The voice processing unit 104 includes an orientation for receiving external sound (analog signal). Figure 1A The microphone on the right side, 19L, and facing... Figure 1A The microphone 19R on the left side generates a speech signal by processing the analog signal received as sound.

[0228] By emitting light, sound, or vibration, the LED 17, speaker 105, or vibrator 106 provides the user with notifications or warnings about the status of the camera body 1.

[0229] The angular velocity sensor 107 is a sensor that uses a gyroscope or the like, and detects the movement of the camera body 1 as gyroscope data.

[0230] Accelerometer 108 detects the posture of camera / detection unit 10.

[0231] An angular velocity sensor 107 and an acceleration sensor 108 are built into the camera / detection unit 10, and an angular velocity sensor 807 and an acceleration sensor 808, which are separately arranged from the angular velocity sensor 107 and the acceleration sensor 108, are also arranged inside the display device 800 described below.

[0232] Figure 6 This is a block diagram illustrating the hardware structure of the display device 800. For simplicity, the reference diagram is omitted. Figure 1D The same reference numerals will be used for the parts described above, and their descriptions will be omitted.

[0233] refer to Figure 6The display device 800 includes a display device control unit 801, button A 802, display unit 803, button B 804, built-in camera 805, face sensor 806, angular velocity sensor 807, acceleration sensor 808, camera signal processing circuit 809, and various switches 811.

[0234] In addition, the display device 800 includes a built-in non-volatile memory 812, a main memory 813, a large-capacity non-volatile memory 814, a speaker 815, a vibrator 816, an LED 817, a voice processing unit 820, a low-power radio unit 871, and a high-speed radio unit 872.

[0235] The display device control unit 801 is composed of a CPU and is connected to a reference. Figure 1D The button A802 and the face sensor 806, etc., are used to control the display device 800.

[0236] Although the image signal processing circuit 809 performs functions equivalent to those of the image driver 41, solid-state image sensor 42, and image signal processing circuit 43 arranged inside the camera body 1, its description is less important in this embodiment. Therefore, for simplicity, they will be described as one. The data output by the image signal processing circuit 809 is processed inside the display device control unit 801. The details of this data processing will be described below.

[0237] Although its details will be omitted, Figure 1D The various switches 811 not shown are switches used to perform functions unrelated to this embodiment.

[0238] The angular velocity sensor 807 is a sensor that uses a gyroscope or the like, and detects the movement of the display device 800.

[0239] Accelerometer 808 detects the posture of display device 800.

[0240] As described above, the angular velocity sensor 807 and the acceleration sensor 808 are integrated into the display device 800 and have functions similar to those of the angular velocity sensor 107 and the acceleration sensor 108 arranged in the camera body 1 described above, and are configured separately.

[0241] The built-in non-volatile memory 812 uses flash memory or the like, and stores the startup program of the display device control unit 801 and the setting values ​​of various program modes in the built-in non-volatile memory 812.

[0242] The main memory 813 is constructed using RAM or the like and temporarily stores video data during processing, as well as the results of arithmetic operations performed by the camera signal processing circuit 809. In this embodiment, during the recording of motion picture video, gyroscope data detected by the angular velocity sensor 807 at the recording time of each frame is stored in the main memory 813 in association with each frame.

[0243] The large-capacity non-volatile memory 814 records or reads image data from the display device 800. In this embodiment, the large-capacity non-volatile memory 814 is constructed using an attachable / removable memory such as an SD card. Similar to the large-capacity non-volatile memory 51 arranged in the camera body 1, the large-capacity non-volatile memory 814 may be constructed using a non-attachable / non-removable memory.

[0244] By emitting sound, vibration, or light, the speaker 815, vibrator 816, or LED 817 provides the user with notifications or warnings regarding the status of the display device 800.

[0245] The voice processing unit 820 includes a left microphone 819L and a right microphone 819R that receive external sound (analog signal) and generates a voice signal by processing the received analog signal as sound.

[0246] The low-power radio unit 871 uses a low-power radio to exchange data with the camera body 1.

[0247] The high-speed wireless unit 872 uses high-speed wireless to exchange data with the camera body 1.

[0248] The face sensor 806 (face detection unit) includes an infrared LED lighting circuit 821, an infrared LED 822, an infrared focusing lens 826, and an infrared detection processing device 827.

[0249] The infrared LED lighting circuit 821 has the characteristics of... Figure 5 The circuit 21 shown is similar in function to the infrared LED lighting circuit 21, controlling the lighting / extinguishing of the infrared LED 822 and controlling the projection of infrared rays 823 from the infrared LED 822 to the user.

[0250] The infrared focusing lens 826 is a lens that focuses the reflected beam 825 of infrared light 823.

[0251] The infrared detection processing device 827 includes a sensor that detects the reflected beam of light focused by the infrared focusing lens 826. The sensor converts the focused reflected beam 825 into sensor data and transmits the sensor data to the display device control unit 801.

[0252] when Figure 1DWhen the face sensor 806 shown is facing the user, such as Figure 6 As shown, infrared rays 823 projected from the infrared LED lighting circuit 821 are emitted onto the infrared emitting surface 824, which serves as the user's entire face. Furthermore, the infrared rays 823 reflected on the infrared emitting surface 824 become a reflected beam 825, which is then focused by the infrared focusing lens 826 onto a sensor arranged in the infrared detection processing device 827.

[0253] Further details of functional unit 830 will be omitted, and the functional unit performs functions that are not related to this embodiment and can be performed by a smartphone alone, such as telephone functions and functions of other sensors.

[0254] The following will explain how to use the camera body 1 and the display device 800.

[0255] Figure 7A This is a flowchart showing an overview of the video recording processing performed in the camera body 1 and the display device 800 according to this embodiment.

[0256] As an aid to explanation, in Figure 7A The text on the right side of each step indicates the step being performed. Figure 4 One of the devices shown. In other words, Figure 7A Steps S100 to S700 shown are performed by the camera body 1, and Figure 7A The steps S800 to S1000 shown are performed by the display device 800.

[0257] In step S100, when the power switch 11 is turned on and power is supplied to the camera body 1, the overall control CPU 101 starts operating and reads the startup program from the built-in non-volatile memory 102. Afterwards, the overall control CPU 101 performs preparatory operations for setting up the camera body 1 before it begins recording. The following will refer to... Figure 7B To explain the details of the preparation and processing.

[0258] In step S200, the face orientation detection unit 20 detects the face orientation, thereby performing face orientation detection processing for estimating the observation orientation. The following will refer to... Figure 7C To explain the details of the face orientation detection process, this process is performed at a predetermined frame rate.

[0259] In step S300, the recording direction / viewpoint determination unit 30 performs recording direction / range determination processing. The following will refer to... Figure 7D This section explains the details of the process for determining the recording direction / range.

[0260] In step S400, the camera unit 40 performs video recording and generates captured image data.

[0261] In step S500, the image cutting / development processing unit 50 performs recording range development processing. In this recording range development processing, the recording direction and viewing angle information determined in step S300 are used to cut out video from the captured image data generated in step S400, and development processing is performed on that range. The following will refer to... Figure 7E This will explain the details of the recording range imaging process.

[0262] In step S600, master recording processing is performed. In this master recording processing, the master recording unit 60 (video recording unit) stores the video after display in step S500 as video data in the main memory 103. The following will refer to... Figure 14 This will explain the details of the master record processing.

[0263] In step S700, a transmission process to the display device 800 is performed. In this transmission process, the transmission unit 70 wirelessly transmits the video recorded in step S600 to the display device 800 at a specified time. The following will refer to... Figure 16 To explain the details of the transmission processing to the display device 800.

[0264] Step S800 and subsequent steps are performed by the display device 800.

[0265] In step S800, the display device control unit 801 performs optical correction processing, in which optical correction is performed on the video transmitted from the camera body 1 in step S700. The following will refer to... Figure 17 To explain the details of the optical correction process.

[0266] In step S900, the display device control unit 801 performs image stabilization processing on the optically corrected video from step S800. The following will refer to... Figure 19 This will explain the details of image stabilization processing.

[0267] The order of steps S800 and S900 can be reversed. In other words, optical correction can be performed later after image stabilization correction of the video is performed first.

[0268] In step S1000, the display device control unit 801 (motion image recording unit) performs auxiliary recording and ends the process. In this auxiliary recording, the video that has completed optical correction and image stabilization processing in steps S800 and S900 is recorded in a large-capacity non-volatile memory 814.

[0269] Next, refer to Figures 7B to 7F In addition, refer to other figures, etc. Figure 7A The subroutines for each step, along with the order of processing, are described in detail.

[0270] Figure 7B yes Figure 7A The flowchart shows the subroutine for the preparation operation process in step S100. In the following text, it will be used... Figures 2A to 2F and Figure 5 The part shown illustrates this process.

[0271] In step S101, it is determined whether the power switch 11 is turned on. If the power is still off, the process is in standby mode. When the power is turned on, the process proceeds to step S102.

[0272] In step S102, the mode selected by the camera mode switch 12 is determined. As a result of this determination, if the mode selected by the camera mode switch 12 is the motion picture mode, the process proceeds to step S103.

[0273] In step S103, various settings for the motion picture mode are read from the built-in non-volatile memory 102 and stored in the main memory 103, and then the process proceeds to step S104. Here, among the various settings for the motion picture mode are image stabilization levels such as "strong", "medium" or "off" specified by the viewpoint setting value ang (in this embodiment, it is preset to 90°).

[0274] In step S104, after the operation of the camera driver 41 for the motion picture mode begins, the process exits from the subroutine.

[0275] As a result of the judgment in step S102, if the mode selected by the camera mode switch 12 is still image mode, the process proceeds to step S106.

[0276] In step S106, various settings for the still image mode are read from the built-in non-volatile memory 102 and stored in the main memory 103, and then the process proceeds to step S107. Here, among the various settings for the still image mode are image stabilization levels such as "strong", "medium" or "off" specified by the viewing angle setting value ang (in this embodiment, it is preset to 45°).

[0277] In step S107, after the operation of the camera driver 41 for still image mode begins, the process exits from the subroutine.

[0278] As a result of the determination in step S102, if the mode selected by the camera mode switch 12 is the Pri setting mode, the process proceeds to step S108. Here, the Pri setting mode is the mode in which the camera body 1 is set from an external device such as the display device 800, and it is one of three camera modes that can be switched using the camera mode switch 12. In other words, the Pri setting mode is a mode used for customizing video recording. Here, the camera body 1 is a small wearable device, therefore, there are no operation switches or setting screens for changing its detailed settings in the camera body 1, and the detailed settings of the camera body 1 are changed by an external device such as the display device 800.

[0279] For example, consider a scenario where, when capturing the same moving image, it is desirable to continue filming with both 90° and 110° viewing angles. Since the normal moving image mode uses a 90° viewing angle, in such a case, it is necessary to first change the display device 800 to the camera body 1's settings screen after ending the filming in normal moving image mode, and then switch the viewing angle to 110°. However, this operation on the display device 800 is cumbersome even during any event.

[0280] On the other hand, when the Pri setting mode is preset to capture motion images from a 110-degree angle, after capturing motion images from a 90-degree angle, simply sliding the camera mode switch 12 to "Pri" immediately switches to capturing motion images from a 110-degree angle. In other words, the user does not need to stop the current action and perform the aforementioned cumbersome operation.

[0281] The details set in the Pri settings mode can include not only the viewing angle, but also the image stabilization level specified as "strong", "medium" or "off", as well as speech recognition settings not described in this embodiment.

[0282] In step S108, various settings for the Pri setting mode are read from the built-in non-volatile memory 102 and stored in the main memory 103, then the process proceeds to step S109. Here, the various settings for the Pri setting mode include the viewing angle setting value ang and the image stabilization level specified as "strong", "medium" or "off".

[0283] In step S109, after the operation of the camera driver 41 for setting the Pri mode begins, the process exits from the subroutine.

[0284] Here, we will refer to Figure 13This will explain the various settings of the motion image mode read in step S103.

[0285] Figure 13 This diagram shows the menu screen used for various settings of the motion image mode displayed on the display unit 803 of the display device 800 before the camera body 1 takes a picture. For... Figure 1D The same reference numerals will be used for the same parts, and their descriptions will be omitted. The display unit 803 has a touch panel function and will be described as operating according to touch operations, including swiping operations.

[0286] refer to Figure 13 The menu screen includes a preview screen 831, a zoom lever 832, a recording start / stop button 833, a switch 834, a battery remaining display 835, a button 836, a lever 837, and an icon display unit 838.

[0287] The preview screen 831 can be used to check the video captured by the camera body 1, and can also be used to check the zoom level and angle of view.

[0288] The zoom lever 832 is an operating unit that can be used to set the zoom by shifting the zoom lever 832 to the left or right. In this embodiment, although the cases in which the four values ​​of 45°, 90°, 110° and 130° can be set as the viewing angle setting value ang are described, values ​​other than these values ​​can be configured to be set as the viewing angle setting value ang by the zoom lever 832.

[0289] The record start / stop button 833 is a toggle switch that functions as both a start switch 14 and a stop switch 15.

[0290] Switch 834 is a switch used to toggle between "off" and "on".

[0291] Battery Remaining Power Display 835 displays the remaining battery level of the camera body 1.

[0292] Button 836 is used to enter other modes.

[0293] Rod 837 is a rod for setting the image stabilization intensity. In this embodiment, although only "strong" and "medium" can be set as the image stabilization intensity, other image stabilization intensities (e.g., "weak") can be configured to be set. Additionally, the image stabilization intensity can be configured to be set in a non-step manner.

[0294] The icon display unit 838 displays multiple thumbnail icons used for the preview.

[0295] Figure 7C yes Figure 7AThe flowchart of the face orientation detection processing subroutine in step S200 is shown. Before explaining the details of this processing, please refer to... Figures 8A to 8K This will illustrate a method for detecting facial orientation using infrared projection.

[0296] Figure 8A This is a diagram showing the user's video as seen from the face-oriented detection window 13.

[0297] In the case where there is no visible light cutoff filter component in the face direction detection window 13, visible light is fully transmitted, and the infrared detection processing device 27 is a visible light imaging element, Figure 8A The video shown is the same as the video captured by a camera element using visible light.

[0298] exist Figure 8A The video shown depicts the front of the neck 201 above the user's collarbone, the base of the jaw 202, the tip of the jaw 203, and the face 204 including the nose.

[0299] Figure 8B This is a diagram showing the situation where fluorescent lights in an indoor location are displayed as a background in a video of a user seen from the face detection window 13.

[0300] exist Figure 8B The video shown depicts multiple fluorescent lights 205 present near the user. Thus, depending on the usage conditions, various backgrounds are displayed in the infrared detection processing unit 27, making it difficult for the face orientation detection unit 20 and the overall control CPU 101 to separate the video of the face portion from the sensor data supplied from the infrared detection processing unit 27. Recently, although techniques exist for separating such videos using AI, these require high capabilities from the overall control CPU 101, and therefore such techniques are unsuitable for the camera body 1 of a portable device.

[0301] In fact, the face orientation detection window 13 is constructed using a visible light cutoff filter and transmits almost no visible light; therefore, the video from the infrared detection processing device 27 and... Figure 8A and Figure 8B The videos shown are different.

[0302] Figure 8C This diagram illustrates video acquired under the following conditions: with the infrared LED 22 off, the infrared detection processing device 27 uses the sensor via the face direction detection window 13 to... Figure 8B The image shows the user and the fluorescent light as its background.

[0303] exist Figure 8CIn the video shown, the user's neck and jaw are darkened. On the other hand, the fluorescent lamp 205 emits not only visible light but also an infrared component, and is therefore shown as slightly brighter.

[0304] Figure 8D This diagram illustrates video acquired under the following conditions: with infrared LED 22 illuminated, the infrared detection processing device 27 uses a sensor via face direction detection window 13 to... Figure 8B The image shows the user and the fluorescent light as its background.

[0305] exist Figure 8D In the video shown, unlike Figure 8C The user's neck and jawline became brighter. On the other hand, there was no change in brightness or other aspects around fluorescent lamp 205.

[0306] Figure 8E It shows the basis Figure 8C and Figure 8D The image shown is a graph of the differential video calculated from the video. It can be understood that the user's face has been revealed.

[0307] Thus, by calculating the difference between the video images captured by the sensor of the infrared detection processing device 27 when the infrared LED 22 is lit and when it is turned off, the overall control CPU 101 (video acquisition unit) acquires the differential video (hereinafter also referred to as facial video) of the user's face.

[0308] According to this embodiment, the facial orientation detection unit 20 employs a system that acquires facial video by extracting infrared reflection intensity as a two-dimensional image using an infrared detection processing device 27. The sensor of the infrared detection processing device 27 adopts a structure similar to that of a general camera element, and acquires one frame of the facial image at a time. A vertical synchronization signal (hereinafter referred to as signal V) for frame synchronization is generated by the infrared detection processing device 27 and output to the overall control CPU 101.

[0309] Figures 9A to 9E This is a timing diagram showing the on / off timing of the infrared LED 22.

[0310] Figure 9A The timing of signal V generated by infrared detection processing device 27 is shown. The timing of frame synchronization and the lighting / extinguishing of infrared LED 22 are measured according to the signal V changing to Hi (high).

[0311] exist Figure 9A In the diagram, t1 represents the time period for the first facial image acquisition, and t2 represents the time period for the second facial image acquisition. Figures 9A to 9D In this context, the time axis on the horizontal axis is represented in the same way.

[0312] exist Figure 9B In the diagram, the vertical axis represents the position H of the image signal output from the sensor of the infrared detection processing device 27. The infrared detection processing device 27 controls the movement of the sensor, such that... Figure 9B As shown, the position H of the image signal is synchronized with the signal V. The sensor of the infrared detection processing device 27 adopts a structure similar to that of a general imaging element, as described above, and its movement is known; therefore, detailed control instructions will be omitted.

[0313] Figure 9C This illustrates the high / low level switching timing of the IR-ON (IR-on) signal output from the overall control CPU 101 to the infrared LED lighting circuit 21. For example... Figure 9C As shown, the high / low level switching of the signal IR-ON is controlled by the overall control CPU 101, making the signal synchronized with the signal V. More specifically, the overall control CPU 101 outputs the low level signal IR-ON to the infrared LED lighting circuit 21 during the time period t1, and outputs the high level signal IR-ON to the infrared LED lighting circuit 21 during the time period t2.

[0314] Here, during the period when the signal IR-ON is high, the infrared LED lighting circuit 21 illuminates the infrared LED 22 and projects infrared light 23 towards the user. On the other hand, during the period when the signal IR-ON is low, the infrared LED lighting circuit 21 turns off the infrared LED 22.

[0315] Figure 9D This is the image data captured by the sensor output from the infrared detection processing device 27 to the overall control CPU 101. The vertical direction represents the signal strength and the amount of light received by the reflected beam 25. In other words, during the time period t1, the infrared LED 22 is off, thus creating a state where there is no reflected beam 25 emitted from the user's face, and the image data is captured as shown in the image. Figure 8C The image data shown is as follows. On the other hand, during the time period t2, the infrared LED 22 is illuminated, thus creating a state where a reflected beam 25 is emitted from the user's face, and data is acquired as shown... Figure 8D The image data shown is from the captured image. For this reason, such as... Figure 9D As shown, the signal strength during time period t2 increased compared to the signal strength during time period t1 due to the reflected beam 25 emitted from the user's face.

[0316] Figure 9E By seeking Figure 9D The difference between the image data captured during time period t1 and the image data captured during time period t2 is shown, and as... Figure 8EAs shown, the captured image data was obtained by extracting only the component of the reflected beam 25 sent from the user's face.

[0317] Including references Figures 8C to 8E and Figures 9A to 9E In the state of the operation described above, Figure 7C The face orientation detection process in step S200 is shown in the figure.

[0318] First, in step S201, when the timing V1 of the time period from the signal V output from the infrared detection processing device 27 to t1 begins, the processing proceeds to step S202.

[0319] Next, in step S202, the signal IR-ON is set to a low level and output to the infrared LED lighting circuit 21. Accordingly, the infrared LED 22 is turned off.

[0320] In step S203, the captured image data corresponding to a frame output from the infrared detection processing device 27 during the time period t1 is read, and the read data is temporarily stored in the main memory 103 as frame 1.

[0321] In step S204, when the timing V2 of the time period t2 in the signal V output from the infrared detection processing device 27 changes to t2, the processing proceeds to step S205.

[0322] In step S205, the signal IR-ON is set to a high level and output to the infrared LED lighting circuit 21. Accordingly, the infrared LED 22 is lit.

[0323] In step S206, image data corresponding to a frame output from the infrared detection processing device 27 during the time period t2 is read, and the read data is temporarily stored as frame 2 in the main memory 103.

[0324] In step S207, the signal IR-ON is set to a low level and output to the infrared LED lighting circuit 21. Accordingly, the infrared LED 22 is turned off.

[0325] In step S208, frames 1 and 2 are read from main memory 103, and the difference obtained by subtracting frame 1 from frame 2 is calculated. Figure 9E The light intensity Fn of the component of the reflected beam 25 shown by the user (this is a process commonly referred to as black subtraction).

[0326] In step S209, the neck position (neck rotation center) is extracted from the light intensity Fn.

[0327] First, the overall control CPU 101 (segmentation unit) segments the facial video into reference segments based on light intensity Fn. Figure 8F The description includes multiple distance regions.

[0328] Figure 8F The diagram illustrates the following situation: Figure 8E The intensity of the differential video shown is scaled to the light intensity of the reflected beam 25 of infrared 23 projected onto the user's face and neck, in order to view the distribution of light amount at various locations on the user's face and neck.

[0329] To simplify the explanation, Figure 8F Chart 1 is a diagram of how... Figure 8E The distribution of light intensity of the reflected beam 25 in the facial video shown is divided into regions and plotted in grayscale. For illustration, the Xf axis is taken from the center of the user's neck towards the tip of the jaw.

[0330] exist Figure 8F In Figure 2, the horizontal axis represents Figure 8F The light intensity in Figure 1 is shown on the Xf axis, with the horizontal axis representing the Xf axis. On the horizontal axis, the light intensity becomes stronger towards the right.

[0331] exist Figure 8F In Figure 1, the facial video is divided into six regions (distance regions) 211 to 216 based on light intensity.

[0332] Region 211 is the region with the strongest light intensity and is represented by white as a grayscale value.

[0333] Region 212 is a region with a light intensity slightly lower than that of region 211, and is represented by a very bright gray as a grayscale.

[0334] Region 213 is a region with a lower light intensity than region 212, and is represented by a light gray as a grayscale.

[0335] Region 214 is a region where the light intensity is further lower than that of region 213, and is represented by a medium gray as a grayscale.

[0336] Region 215 is a region where the light intensity is further lower than that of region 214, and is represented by a slightly darker gray as a grayscale.

[0337] Region 216 is the area with the weakest light intensity and is represented by the darkest gray in the grayscale. The upper side of region 216 is represented by black, which has no light intensity.

[0338] In the following text, reference will be made to Figures 10A to 10D Let me explain this light intensity in detail.

[0339] Figures 10A to 10D It is a diagram showing the vertical movement of the user's face and representing the state as observed from the user's left horizontal direction.

[0340] Figure 10A This diagram shows the user's appearance facing forward. The camera / detection unit 10 is located in front of the user's collarbone. Furthermore, infrared rays 23 from an infrared LED 22 are emitted from a facial direction detection window 13 located at the top of the camera / detection unit 10 towards the lower part of the user's head. Given that the distance from the facial direction detection window 13 to the base 200 of the neck above the user's collarbone is represented by Dn, the distance from the facial direction detection window 13 to the base 202 of the jaw is represented by Db, and the distance from the facial direction detection window 13 to the tip 203 of the jaw is represented by Dc, it can be understood that the distances increase in the order of Dn, Db, and Dc. Since light intensity is inversely proportional to the square of the distance, the light intensity acquired when the reflected beam 25 emitted from the infrared emitting surface 24 is imaged in the sensor of the infrared detection processing device 27 decreases in the order of the base 200 of the neck, the base 202 of the jaw, and the tip 203 of the jaw. Furthermore, the light intensity of the face 204, including the nose, located at a position longer than Dc relative to the face orientation detection window 13, is further dimmed. In other words, in such a position... Figure 10A In the case shown, obtain with Figure 8F The video shows the distribution of light intensity.

[0341] The configuration of the face orientation detection unit 20 is not limited to the configuration shown in this embodiment, as long as it can detect the user's face orientation. For example, an infrared pattern can be emitted from the infrared LED 22 (infrared pattern emitting unit), and the infrared pattern reflected from the emitting target can be detected by the sensor (infrared pattern detection unit) of the infrared detection processing device 27. In this case, it is preferable that the sensor of the infrared detection processing device 27 is a structured optical sensor. Alternatively, the sensor of the infrared detection processing device 27 can be a sensor that compares the phases of the infrared light 23 and the reflected beam 25 (infrared phase comparison unit) (e.g., a ToF sensor).

[0342] Next, refer to Figure 8G To explain Figure 7C The neck position is extracted in step S209 shown.

[0343] Figure 8G Figure 1 is the following diagram, in which... Figure 8F Superposition representation Figure 10A The text shows markings for various parts of the user's body, as well as double circles and black circles indicating the neck and chin positions, respectively.

[0344] White area 211 and base of neck 200 ( Figure 10A Correspondingly, the very bright gray area 212 corresponds to the front of the neck 201. Figure 10A Corresponding to this, and the light gray area 213 corresponds to the base 202 of the jaw. Figure 10A Corresponding to ). Additionally, the medium gray area 214 corresponds to the cusp 203 ( Figure 10A Corresponding to, and slightly darker gray area 215 is located on face 204 ( Figure 10A The lower part of the lips corresponds to the lower part of the face present nearby. Furthermore, the dark gray area 216 corresponds to the area located on the face 204 ( Figure 10A The nose in the center corresponds to the upper part of the face that is located nearby.

[0345] like Figure 10A As shown, the difference between distances Db and Dc is small relative to the distance from the detection window 13 in the face direction to other parts of the user, so the difference in reflected light intensity between the light gray area 213 and the medium gray area 214 is also small.

[0346] On the other hand, such as Figure 10A As shown, among the distances from the detection window 13 in the face direction to various parts of the user, the distance Dn is the shortest. Therefore, the white area 211 corresponding to the base 200 of the neck is the place with the strongest reflection intensity.

[0347] Therefore, the overall control CPU 101 (setting unit) will be located near the base 200 of the neck of region 211, at the horizontal center of region 211, closest to the camera / detection unit 10 (in... Figure 8G Position 206 (represented by double circles in Figure 1) is set as the position of the neck rotation center (hereinafter referred to as neck position 206). The processing up to this point is represented in... Figure 7C The details of step S209 shown are as follows.

[0348] Next, refer to Figure 8G To explain Figure 7C The extraction of the cusp position in step S210 is shown.

[0349] Figure 8G The medium gray area 214 shown in Figure 1, which is brighter than the area 215 corresponding to the lower part of the face including the lips in face 204, is the area including the chin tip. (As shown from...) Figure 8GAs can be understood from Figure 2, in region 215, which is in contact with region 214, the light intensity drops sharply, and the distance change relative to the facial orientation detection window 13 increases. The overall control CPU 101 identifies region 214 anterior to region 215, where the light intensity drops sharply, as the apex region. Additionally, the overall control CPU 101 calculates (extracts) the position furthest from the neck position 206 at the horizontal center of region 214. Figure 8G The position indicated by the black circle in Figure 1 is the cusp position 207.

[0350] For example, Figure 8H and Figure 8I The changes observed when the face is turned to the right are shown.

[0351] Figure 8H This is shown when the user's face is facing right. Figure 8E The diagram shows a differential video calculated using a method similar to the one shown. Figure 8I By superimposing, they are respectively represented as Figure 8H The image obtained by marking the double circles and black circles at the center position of the neck operation, cervical position 206 and chin tip position 207r.

[0352] Since the user is facing right, area 214 appears to the left when viewed from the side of the camera / detection unit 10. Figure 8I The region 214r represented in the image moves. The region 215 in the face 204 corresponding to the lower part of the face, including the lips, also moves to the region 215r that exists in the leftward direction when viewed from the camera / detection unit 10 side upward.

[0353] Therefore, the overall control CPU 101 identifies the region 214r, located anterior to the region 215r where a sharp drop in light intensity occurs, as the apical region. Furthermore, the overall control CPU 101 calculates (extracts) the position further from the neck position 206 at the horizontal center of region 214r. Figure 8I The position indicated by the black circle in the middle is the position of the cusp 207r.

[0354] Then, the overall control CPU 101 acquires the movement angle θr, which represents... Figure 8I The cusp position 207r shown is centered on the neck position 206. Figure 8G The amount of rightward movement of the cusp position 207 in Figure 1. (As shown) Figure 8I As shown, the movement angle θr is the horizontal angle of the user's face.

[0355] By using the above method, in step S210, the infrared detection processing device 27 of the facial orientation detection unit 20 (three-dimensional detection sensor) detects the position of the chin tip and the angle of the horizontal direction of the user's face.

[0356] Next, we will explain the detection of the upward direction of the face.

[0357] Figure 10B It is a diagram showing the appearance of a user with their face oriented horizontally, and Figure 10C It is a diagram showing the appearance of a user with his or her face facing upwards at 33° from the horizontal direction.

[0358] exist Figure 10B In the diagram, the distance from the facial detection window 13 to the jaw apex 203 is represented by Ffh, and... Figure 10C In this context, the distance from the facial detection window 13 to the jaw tip 203u is represented by Ffu.

[0359] like Figure 10C As shown, the cusp 203u also moves upward along with the face, so it can be understood that the distance Ffu is longer than the distance Ffh.

[0360] Figure 8J This is a diagram showing the video of the user as seen from the face direction detection window 13 when the user tilts his or her face upwards at a 33° angle from the horizontal. (See diagram 13 for example.) Figure 10C As shown, the user is facing upwards, so the facial detection window 13, located below the user's jaw, does not show the face 204, including the lips and nose, but only extends up to the jaw apex 203. At this time, in Figure 8K The diagram shows the distribution of light intensity of the reflected beam 25 when infrared 23 is emitted to the user. Figure 8K It is shown in the following figure: When using and Figure 8E The method shown is similar to the one used to calculate the differential video, which is superimposed with double circles and black circles to represent the neck position 206 and the chin position 207u, respectively.

[0361] and Figure 8K The six regions 211u to 216u corresponding to the light intensity shown are obtained by appending "u" to the corresponding light intensity. Figure 8F The regions shown represent areas with the same light intensity as the regions with the same light intensity. It can be understood that, although in Figure 8F The light intensity of the cusp 203 of the middle user exists in the mid-gray region 214, but... Figure 8K The light intensity shifts towards the gray side and exists in the slightly darker gray region 215u. Thus, as... Figure 10C As shown, it can be detected by the infrared detection and processing device 27 that the distance Ffu is longer than the distance Ffh. As a result, the light intensity of the reflected beam 25 of the user's jaw tip 203 weakens and is inversely proportional to the square of the distance.

[0362] Next, we will explain the detection of the downward direction of the face.

[0363] Figure 10D It is a diagram showing the appearance of a user oriented his or her face at a direction 22° lower than the horizontal direction.

[0364] exist Figure 10D In this context, the distance from the facial detection window 13 to the jaw tip 203d is represented by Ffd.

[0365] like Figure 10D As shown, the cusp 203d also moves downward along with the face, so it can be understood that the distance Ffd is shorter than the distance Ffh, and the light intensity of the reflected beam 25 of the cusp 203 becomes stronger.

[0366] Return to reference Figure 7C In step S211, the overall control CPU 101 (distance calculation unit) uses the light intensity at the chin tip position detected by the infrared detection processing device 27 of the facial direction detection unit 20 (three-dimensional detection sensor) to calculate the distance from the chin tip position to the facial direction detection window 13. Based on this, the angle of the face in the vertical direction is also calculated.

[0367] In step S212, the angles of the horizontal direction (first detection direction) and the vertical direction perpendicular to the horizontal direction (second detection direction) of the face obtained in steps S210 and S211 are stored in the main memory 103 as the user's three-dimensional viewing direction vi (here, i is an arbitrary symbol). For example, since the horizontal direction θh is 0° and the vertical direction θv is 0°, the viewing direction vo when the user is observing the center of the front becomes vector information of [0°, 0°]. In addition, the viewing direction vr when the user is observing the right side at 45° becomes vector information of [45°, 0°].

[0368] In step S211, although the angle in the vertical direction of the face is calculated by detecting the distance relative to the face direction detection window 13, the calculation method is not limited to this. For example, the angle change can be calculated by comparing the level of change in light intensity at the chin tip 203. In other words, for the distance relative to the face direction detection window 13... Figure 8G The gradient CDh of the reflected light intensity of the jaw tip 203 relative to the base 202 of the jaw in Figure 2 can be based on the relative... Figure 8K The angular change of the jaw is calculated by the gradient change of the reflected light intensity CDu of the jaw tip 203 with respect to the base 202 of the jaw in Figure 3.

[0369] Figure 7D yes Figure 7AThe flowchart of the recording direction / range determination subroutine for step S300 is shown. Before explaining the details of this process, please refer to... Figure 11A This describes an ultra-wide-angle video as an object for determining the recording direction and recording range according to this embodiment.

[0370] In the camera body 1 according to this embodiment, the camera unit 40 uses a camera lens 16 with an ultra-wide angle to capture an ultra-wide angle video near the camera / detection unit 10 and cut out a portion of it, thereby realizing the acquisition of video in the direction of observation.

[0371] Figure 11A This is a diagram showing the target field of view 125 of the ultra-wide-angle video captured by the camera unit 40 when the user is facing forward.

[0372] like Figure 11A As shown, the pixel area 121 that can be captured by the solid-state camera element 42 is a rectangular area. The effective projection section 122 (predetermined area) is the area where the circular hemispherical video projected onto the solid-state camera element 42 by the camera lens 16 is displayed. The camera lens 16 is adjusted so that the center of the pixel area 121 and the center of the effective projection section 122 are aligned with each other.

[0373] The outermost periphery of the effective projection section 122, which has a circular shape, represents a position with a field of view (FOV) angle of 180 degrees. When the user is viewing the horizontal and vertical center, the angle of the target field of view 125, which is the area captured and recorded from the center of the effective projection section 122, is 90°, which is half the FOV angle. According to this embodiment, the camera lens 16 can also introduce a light beam outside the effective projection section 122, and can perform fisheye projection of a light beam with a maximum FOV angle of up to about 192° onto the solid-state imaging element 42. However, when it goes beyond the effective projection section 122, the resolution deteriorates drastically, the light intensity decreases, the distortion becomes stronger, or the optical performance deteriorates greatly. Therefore, in this embodiment, an example will be described as follows: in the recording area, the video in the viewing direction is cut out only from the inside of the video projected onto the pixel area 121 from the hemispherical video displayed in the effective projection section 122 (hereinafter referred to as the ultra-wide-angle video).

[0374] In this embodiment, since the size of the effective projection portion 122 in the vertical direction is larger than the size of the short side of the pixel region 121, the video at the upper and lower ends of the effective projection portion 122 is offset from the pixel region 121. However, this configuration is not limited to this. For example, by changing the configuration of the camera lens 16, all effective projection portions 122 can be designed to enter the interior of the pixel region 121.

[0375] Invalid pixel region 123 is a pixel region in pixel region 121 that is not included in the effective projection section 122.

[0376] The target field of view 125 represents the area of ​​the video cut out from the ultra-wide-angle image in the user's viewing direction, and is defined using pre-set vertical and horizontal viewing angles centered on the viewing direction (here, 45°, field of view angle 90°). Figure 11A In the example shown, the user is facing forward, so the center of the target field of view 125 is in the observation direction vo, which is the center of the effective projection unit 122.

[0377] exist Figure 11A The ultra-wide-angle video shown includes subject A131 as a child, subject B132 as the steps that the child, as subject A, is about to climb, and subject C133 as a play tool in the form of a train.

[0378] Next, in Figure 7D The above is shown in the figure. Figure 11A The recording direction / range determination process in step S300, which involves acquiring video of the observation direction in the ultra-wide-angle video, is described below. A specific example of the target field of view 125 will be referenced in the following text. Figures 12A to 12G This will explain the process.

[0379] In step S301, the view setting value ang is obtained by reading the preset view setting value ang from the main memory 103.

[0380] In this embodiment, all the viewing angles of 45°, 90°, 110°, and 130° used by the image cropping / display processing unit 50 to crop the video of the viewing direction from the ultra-wide-angle image are stored as viewing angle setting values ​​ang in the built-in non-volatile memory 102. Additionally, in one of steps S103, S106, and S108, one of the viewing angle setting values ​​ang stored in the built-in non-volatile memory 102 is set and stored in the main memory 103.

[0381] In addition, in step S301, the observation direction vi determined in step S212 is determined as the recording direction, and the video of the target field of view 125 cut from the ultra-wide-angle video with the determined direction as the center and the angle setting value ang obtained above is stored in the main memory 103.

[0382] For example, setting the viewing angle value ang to 90° and performing detection processing in the face direction ( Figure 7C If the observation direction vo (vector information [0°, 0°]) is detected in the effective projection unit 122, the range of 45° horizontally and 45° vertically centered on the center O of the effective projection unit 122 is set as the target field of view 125. Figure 11AIn other words, the overall control CPU 101 (relative position setting unit) sets the angle of the facial direction detected by the facial direction detection unit 20 as the viewing direction vi, which represents vector information of the relative position with respect to the ultra-wide-angle video.

[0383] Here, with the observation direction vo in mind, most of the influence of optical distortion due to the camera lens 16 can be ignored. Therefore, the shape of the set target field of view 125 remains unchanged after the distortion conversion in step S303, which will be explained below. Figure 12A The shape of the target field of view 125 after distortion conversion in the case of the observation direction vi will be referred to as the target field of view 125i.

[0384] Next, in step S302, the image stabilization level is obtained by reading the preset image stabilization level from the main memory 103.

[0385] In this embodiment, as described above, in one of steps S103, S106 and S108, the image stabilization level is set and stored in the main memory 103.

[0386] In addition, in step S302, the number of prepared pixels for image stabilization Pis is set based on the obtained image stabilization level.

[0387] In image stabilization processing, a video is acquired based on the deviation of the camera / detection unit 10, using the video in the direction opposite to the deviation direction. For this reason, in this embodiment, a preparatory area required for image stabilization is provided near the target field of view 125i.

[0388] Furthermore, in this embodiment, a table is stored in the built-in non-volatile memory 102, which associates the value of the image stabilization preparation pixel number Pis with each image stabilization level. For example, when the image stabilization level is "intermediate", a preparation pixel region of 100 pixels, which is the image stabilization preparation pixel number Pis read from the above table, is set as a preparation region.

[0389] Figure 12E It is shown in Figure 12A The diagram shows an example of a preparatory area assigned near the target field of view of 125°. Here, the case where the image stabilization level is "intermediate" (in other words, the image stabilization preparatory pixel amount Pis is 100 pixels) will be explained.

[0390] like Figure 12E As shown, the dashed portion with a margin of 100 pixels (preparation area) as the number of reserved pixels Pis for image stabilization on the upper, lower, left and right sides of the target field of view 125o is called the reserved pixel box 126o for image stabilization.

[0391] exist Figure 12A and Figure 12E For simplicity, the example described above assumes that the observation direction vi coincides with the center O (the center of the optical axis of the camera lens 16) of the effective projection section 122. However, as described in the following steps, when the observation direction vi is located at the periphery of the effective projection section 122, optical distortion occurs, necessitating a conversion.

[0392] In step S303, the shape of the target field of view 125 set in step S301 is corrected (distortion conversion) taking into account the observation direction vi and the optical characteristics of the camera lens 16, and a target field of view 125i is generated. Similarly, the image stabilization set in step S302 is also corrected using a pre-set number of pixels Pis, taking into account the observation direction vi and the optical characteristics of the camera lens 16.

[0393] For example, suppose the viewing angle setting value ang is 90°, and the user is observing at 45° to the right relative to the center o. In this case, the observation direction vi determined in step S212 is the observation direction vr (vector information [45°, 0°]), and the range of 45° horizontal and 45° vertical centered on the observation direction vr becomes the target field of view 125. However, considering the optical characteristics of the camera lens 16, the target field of view 125 is corrected to... Figure 12B The target field of view shown is 125r.

[0394] like Figure 12B As shown, the target field of view 125r widens towards the periphery of the effective projection portion 122, and the position of the observation direction vr is also slightly inside the center of the target field of view 125r. This is because, in this embodiment, the camera lens 16 is optically designed to approximate a stereoscopic projection fisheye. When the camera lens 16 is designed based on an equidistant projection fisheye, an equal solid angle projection fisheye, or an orthographic projection fisheye, this relationship changes, and the target field of view 125 is corrected according to its optical characteristics.

[0395] Figure 12F The following diagram illustrates an example: In Figure 12B The target field of view 125r shown is assigned to the vicinity with Figure 12E The prepared region shown has the same image stabilization level "middle" as the prepared region.

[0396] Image stabilization uses a pre-defined pixel box of 126°. Figure 12E In the target field of view 125°, a margin of 100 pixels is set on each of the top, bottom, left, and right sides, corresponding to the number of image stabilization reserve pixels Pis. Conversely, in the image stabilization reserve pixel frame 126r ( Figure 12FIn the image stabilization preparation pixel count Pis, the number of pixels Pis is corrected to increase towards the periphery of the effective projection portion 122.

[0397] Thus, as in Figure 12F As shown in the image stabilization preparation pixel frame 126r, similar in shape to the target field of view 125r, the shape of the preparation area required for image stabilization arranged nearby also has a larger correction amount towards the periphery of the effective projection portion 122. This is because, in this embodiment, the camera lens 16 is optically designed to approximate a stereoscopic projection fisheye. When the camera lens 16 is designed based on an equidistant projection fisheye, an equal solid angle projection fisheye, or an orthographic projection fisheye, this relationship changes, and the image stabilization preparation pixel frame 126r is corrected according to optical characteristics.

[0398] The process of sequentially switching the shapes of the target field of view 125 and the preparation area in step S303, taking into account the optical characteristics of the camera lens 16, is complex. For this reason, in this embodiment, step S303 is performed using a table stored in the built-in non-volatile memory 102 containing the shapes of the target field of view 125i and its preparation area for each viewing direction vi. Based on the pre-provided optical design of the camera lens 16, the optical distortion value can be calculated using a formula included in the overall control CPU 101.

[0399] In step S304, the position and size of the video recording range are calculated.

[0400] As described above, in step S303, a preparatory region required for image stabilization is arranged near the target field of view 125i, and this preparatory region is calculated as a preparatory pixel frame 126i for image stabilization. However, depending on the position of the viewing direction vi, for example, as in the preparatory pixel frame 126r for image stabilization, the shape becomes very specific.

[0401] The overall control CPU 101 performs image processing only on areas of this specific shape, thereby cutting out video. However, when recording video as video data in step S600 or sending video to the display device 800 in step S700, videos with non-rectangular shapes are generally not used. Therefore, in step S304, the position and size of the rectangular video recording frame 127i, which includes the entire image stabilization preparation pixel frame 126i, are calculated.

[0402] Figure 12F The video recording frame 127r, represented by a dotted line, is shown in step S304, calculated using the pre-pixel frame 126r for image stabilization.

[0403] In step S305, the position and size of the video recording frame 127i calculated in step S304 are recorded in the main memory 103.

[0404] In this embodiment, the coordinates Xi and Yi of the upper left side of the video recording frame 127i in the ultra-wide-angle video are recorded as the position of the video recording frame 127i, and the horizontal width WXi and vertical width WYi of the video recording frame 127i obtained from the coordinates Xi and Yi are recorded as the size of the video recording frame 127i. For example, for Figure 12F The video recording frame 127r shown records the coordinates Xr and Yr, horizontal width WXr, and vertical width WYr illustrated in step S305. The coordinates Xi and Yi are XY coordinates with a predetermined reference point (more specifically, with the optical center of the camera lens 16 as the origin).

[0405] Thus, with the image stabilization pre-pixel frame 126i and the video recording frame 127i determined, processing from... Figure 7D The subroutine shown in step S300 exits.

[0406] In the explanations so far, to simplify the description of the complex optical distortion conversion, an observation direction vi has been used as an example, including a horizontal 0° observation direction (that is, observation direction vo (vector information [0°, 0°]) and observation direction vr (vector information [45°, 0°])). However, in reality, the user's observation direction vi can be various directions. Therefore, the recording range imaging processing performed in this case will be explained below.

[0407] For example, with the viewing angle setting ang set to 90° and the observation direction vl set to [-42°, -40°], the target field of view 125l is as follows: Figure 12C As shown.

[0408] Additionally, under the same viewing direction vl (vector information [-42°, -40°]) as the target field of view 125l, when the viewing angle setting ang is 45°, as... Figure 12D As shown, a target field of view 128l is formed that is much smaller than the target field of view 125l. Furthermore, for the target field of view 128l, as... Figure 12G As shown, the image stabilization pre-pixel frame 129l and the video recording frame 130l are set.

[0409] Step S400 is the basic operation of image capture. The imaging unit 40 uses a general sequence, so details can be found in other literature and will be omitted here. In this embodiment, the imaging signal processing circuit 43 arranged in the imaging unit 40 also performs processing to correct the signal from the solid-state imaging element 42 in a unique output format (examples of specifications: MIPI and SLVS) to the captured image data of a general sensor readout system.

[0410] When the mode selected by the camera mode switch 12 is the moving image mode, the camera unit 40 begins recording upon pressing the start switch 14. Recording then ends when the stop switch 15 is pressed. Conversely, when the mode selected by the camera mode switch 12 is the still image mode, the camera unit 40 captures a still image each time the start switch 14 is pressed.

[0411] Figure 7E yes Figure 7A The flowchart shows the subroutine of the recording range imaging process in step S500.

[0412] In step S501, the raw data of the entire area of ​​the captured image data (ultra-wide-angle video) generated by the camera unit 40 in step S400 is obtained, and the raw data is input to the video capture unit (not shown in the figure) of the head, which is called the overall control CPU 101.

[0413] Next, in step S502, based on the coordinates Xi and Yi, horizontal width WXi, and vertical width WYi recorded in the main memory 103 in step S305, a portion of the video recording frame 127i is cut out from the ultra-wide-angle video obtained in step S501. After this cutting out, only for the pixels within the image stabilization preparation pixel frame 126i, the following steps S503 to S508 constitute the cropping and display processing (…). Figure 7F (Start) Accordingly, compared to performing image processing on the entire area of ​​the ultra-wide-angle video read in step S501, the amount of computation can be significantly reduced, and computation time and power consumption can be reduced.

[0414] like Figure 7F As shown, when the mode selected by the camera mode switch 12 is the motion picture mode, the processing of steps S200 and S300 and the processing of step S400 are performed in parallel at the same frame rate or different frame rates. In other words, each time the raw data of the entire area corresponding to a frame generated by the camera unit 40 is acquired, cropping and display processing is performed based on the coordinates Xi and Yi, the horizontal width WXi and the vertical width WYi recorded in the main memory 103 at that time point.

[0415] When the cropping and imaging process for pixels within the pre-defined pixel frame 126i for image stabilization begins, firstly, in step S503, color compensation is performed to supplement the color pixel information configured according to the Bayer array.

[0416] After adjusting the white balance in step S504, color conversion is performed in step S505.

[0417] In step S506, gamma correction is performed to correct the gray scale based on a preset gamma correction value.

[0418] In step S507, edge enhancement is performed based on the image size.

[0419] In step S508, the data is converted into a primarily storable data format and recorded in main memory 103 through processing other than compression, and then the process exits from the subroutine. Details of the primarily storable data format will be described below.

[0420] The order and presence / absence of the cropping and image processing in steps S503 to S508 can be determined according to the camera system and are not limiting to the present invention.

[0421] In addition, if the motion picture mode is selected, the processing steps S200 to S500 are repeated until the recording ends.

[0422] According to this process, the computational load can be significantly reduced compared to the case where imaging processing is performed on the entire area read in step S501. For this reason, a low-power microcomputer can be used as the overall control CPU 101 at low cost, the heat generation of the overall control CPU 101 can be suppressed, and the battery 94's durability can also be improved.

[0423] In addition, in this embodiment, in order to reduce the control load of the overall control CPU 101, optical correction processing for video is performed ( Figure 7A Step S800 shown) and image stabilization processing ( Figure 7AStep S900 shown is not performed by the camera body 1, but by the display device control unit 801 after the video is sent to the display device 800. For this reason, if only the data of the video partially cut out from the projected ultra-wide-angle video is sent to the display device 800, optical correction processing and image stabilization processing cannot be performed. In other words, by using only the data of the cut-out video, there is no position information for substitution during optical correction processing or for reference from the correction table during image stabilization processing; therefore, such processing cannot be performed correctly in the display device 800. For this reason, in this embodiment, not only the data of the cut-out video, but also correction data including information such as the cut-out position of the ultra-wide-angle video is sent from the camera body 1 to the display device 800.

[0424] Here, when the cut-out video is a still image video, and the still image video data and correction data are sent to the display device 800 separately, there is a one-to-one correspondence between the still image video data and the correction data. Therefore, optical correction processing and image stabilization processing can be performed correctly in the display device 800. On the other hand, when the cut-out video is a moving image video, when the moving image video data and correction data are sent to the display device 800 separately, it is difficult to determine whether the sent correction data is the correction data for a specific frame of the moving image video. In particular, if the clock rate of the overall control CPU 101 in the camera body 1 is slightly different from the clock rate of the display device control unit 801 in the display device 800, synchronization between the overall control CPU 101 and the display device control unit 801 cannot be achieved during several minutes of moving image shooting. As a result, the following problem exists: the display device control unit 801 uses correction data that is different from the correction data corresponding to the frame to be processed to correct the frame.

[0425] Therefore, in this embodiment, when the cut-out motion image video data is sent from the camera body 1 to the display device 800, correction data is appropriately assigned to the motion image video data. The method will be described below.

[0426] Figure 14 yes Figure 7A The flowchart of the main record processing subroutine for step S600 is shown below. Reference will be made below. Figure 15 This will explain the process. Figure 14 The process is shown when the mode selected by the camera mode switch 12 is the moving image mode. When the selected mode is the still image mode, the process begins with the process in step S601 and ends when the process in step S606 ends.

[0427] In step S601a, the overall control CPU 101 processes the image within the recording range ( Figure 7E The motion picture video after display is read from the unprocessed frame of the image processed in steps S601 to S606. In addition, the overall control CPU 101 (metadata generation unit) generates correction data as metadata for the read frame.

[0428] In step S601, the overall control CPU 101 appends the cut-out position information of the frame image read in step S600 to the correction data. This appended information is the coordinates Xi and Yi of the video recording frame 127i obtained in step S305. This appended information can be referred to as vector information representing the viewing direction vi.

[0429] In step S602, the overall control CPU 101 (optical correction value acquisition unit) acquires the optical correction value. The optical correction value is the optical distortion value set in step S303. Alternatively, the optical correction value can be a correction value based on lens optical characteristics such as peripheral light amount correction value and diffraction correction.

[0430] In step S603, the overall control CPU 101 adds the optical correction value used for distortion conversion in step S303, which was obtained in step S602, to the correction data.

[0431] In step S604, the overall control CPU 101 determines whether the current mode is an image stabilization mode. More specifically, if the preset image stabilization mode is "medium" or "strong", it is determined to be an image stabilization mode, and the process proceeds to step S605. On the other hand, if the preset image stabilization mode is "off", it is determined that the current mode is not an image stabilization mode, and the process proceeds to step S606. The reason for skipping step S605 when the image stabilization mode is "off" is that skipping this step reduces the amount of data calculated by the overall control CPU 101 and the amount of data transmitted wirelessly, and also reduces the power and heat generation of the camera body 1. Here, although the reduction of data used in image stabilization processing is explained, the data reduction can also be applied to the peripheral light correction value and the presence / absence of analysis correction included in the optical correction value obtained in step S602.

[0432] In this embodiment, although an image stabilization mode is preset based on the operation of the user using the display device 800, the image stabilization mode can be set to the initial setting of the camera body 1. Furthermore, in the case where a camera system is formed that switches between having and not having image stabilization processing after sending video to the display device 800, step S604 is cancelled, and the process proceeds directly from step S603 to step S605.

[0433] In step S605, the overall control CPU 101 (motion detection unit) appends the image stabilization mode obtained in step S302 and the gyroscope data during motion image capture associated with the frame stored in the main memory 813 read in step S600 to the correction data.

[0434] In step S606, the video file 1000 is updated using the image data of the frame read in step S600 and the data obtained by encoding the correction data for which various data were appended in steps S601 to S605. Figure 15 If the first frame of the motion image video is read in step S601a, video file 1000 is generated in step S606.

[0435] In step S607, it is determined that the image processing is within the recording range ( Figure 7E The process checks whether the reading of all frames of the motion picture video after display has finished, and if the reading is not finished, the process returns to step S601a. ​​On the other hand, if the reading is finished, the process exits from this subroutine. The generated video file 1000 is stored in the built-in non-volatile memory 102. The video file can be stored not only in the main memory 813 and the built-in non-volatile memory 102, but also in the large-capacity non-volatile memory 51. In addition, the generated video file 1000 can be immediately sent to the display device 800. Figure 7A The video file can be stored in the main memory 813 after the video file is sent to the display device 800 (as shown in step S700).

[0436] Here, in this embodiment, although the encoding means combining video data and correction data into a single file, the video data can be compressed, or the video data and correction data can be combined and then compressed.

[0437] Figure 15 This is a diagram showing the data structure of video file 1000.

[0438] Video file 1000 consists of header 1001 and frame 1002. Frame 1002 consists of frame datasets, in which the images constituting each frame of the moving image video and the corresponding frame elements are formed into a set. In other words, frame 1002 contains a frame dataset corresponding to the total number of frames in the moving image video.

[0439] In this embodiment, although a frame element is information encoded with correction data, including cutout position (position information within the video), optical correction value, and gyroscope data, as needed, the frame element is not limited to this. For example, the amount of information in a frame element can be changed by adding other information to the frame element according to the camera mode selected by the camera mode switch 12, or by deleting information present in the frame element.

[0440] In the first 1001, the offset value of each frame up to the frame dataset or the starting address of each frame is recorded. Alternatively, metadata such as time and size corresponding to the video file 1000 can be stored.

[0441] Thus, in the main record processing ( Figure 14 In the recording range imaging process, ( Figure 7E The frames of the motion image video displayed by the camera body 1 and their metadata are formed into a video file 1000 and sent to the display device 800. Therefore, even if the clock frequency of the overall control CPU 101 of the camera body 1 is slightly different from the clock frequency of the display device control unit 801 of the display device 800, the display device control unit 801 can reliably perform correction processing on the motion image video displayed by the camera body 1.

[0442] In this embodiment, although optical correction values ​​are included in the frame, the optical correction values ​​can be assigned to the entire video.

[0443] Figure 16 yes Figure 7A The flowchart shows the subroutine of step S700, which involves sending data to the display device 800. Figure 16 The diagram illustrates the processing performed when the mode selected by the camera mode switch 12 is the moving image mode. When the selected mode is the still image mode, the processing begins from step S702.

[0444] In step S701, it is determined whether the recording of motion image video using camera unit 40 (step S400) has ended and whether recording is in progress. Here, when motion image video is being recorded (during motion image capture), a state is formed in which video file 100 is updated (S606) in sequentially performing image processing of the recording range for each frame (step S500) and main recording processing (step S600). Wireless transmission has a high power load, so when wireless transmission is performed in parallel during recording, a large battery capacity of battery 94 is required, or additional heat-fighting measures are needed. In addition, from the perspective of computing power, when wireless transmission is performed in parallel during recording, the computing load increases, so a high-specification overall control CPU 101 is required, and the cost also increases. In this embodiment, taking these factors into consideration, after waiting for the recording of motion image video to end ("Yes" in step S701), the process proceeds to step S702, and a connection with display device 800 is established. However, since the camera system according to this embodiment has a margin in terms of power supplied from the battery 94 and does not require separate countermeasures for heat generation, the connection to the display device 800 can be made in advance when the camera body 1 is turned on or before recording begins.

[0445] In step S702, in order to send the large video file 1000 to the display device 800, a connection is established with the display device 800 via the high-speed wireless unit 72. Although the low-power radio unit 71 is used for transmitting low-resolution video (or video) for checking the viewing angle of the display device 800, and for transmitting / receiving various setting values ​​relative to the display device 800, the low-power radio unit 71 requires transmission time and is therefore not used for transmitting the video file 1000.

[0446] In step S703, the video file 1000 is transmitted to the display device 800 via the high-speed wireless unit 72. At the time when the transmission ends, the process proceeds to step S704, and after the connection with the display device 800 is closed, the process exits from the subroutine.

[0447] So far, while the case of sending a single video file containing all frames of a moving image video has been described, in the case of a long moving image video lasting several minutes, multiple video files segmented by time units can be sent. In the case of having Figure 15 In the case of a video file with the data structure shown, even when a motion picture video is sent to the display device 800 as multiple video files, the motion picture video can be corrected in the display device 800 without causing the timing to deviate from the correction data.

[0448] Figure 17 yes Figure 7A The flowchart of the optical correction process subroutine in step S800 is shown. Reference will be made below. Figures 18A to 18F To illustrate this process, as described above, this process is performed by the display device control unit 801 of the display device 800.

[0449] In step S801, firstly, the display device control unit 801 (video file receiving unit) receives the video file 1000 sent from the camera body 1 during the transmission process to the display device 800 (step S700). Then, the display device control unit 801 (first extraction unit) obtains the optical correction values ​​extracted from the received video file 1000.

[0450] Subsequently, in step S802, the display device control unit 801 (second extraction unit) acquires video (an image of a frame acquired by motion picture capture) from the video file 1000.

[0451] In step S803, the display device control unit 801 (frame video correction unit) uses the optical correction value obtained in step S801 to perform optical correction on the video obtained in step S802, and stores the corrected video in the main memory 813. When performing optical correction, if the video obtained in step S802 is cut out, the cutting process (cutting out the display area) is performed within the range of the image that is narrower than the display range (target field of view 125i) determined in step S303.

[0452] Figures 18A to 18F It is shown in Figure 17 The figure shows the distortion and aberration correction process in step S803.

[0453] Figure 18A This is a diagram showing the position of the subject 1401 as seen by the user's naked eye during filming, and Figure 18B This is a diagram showing an image of the subject 1401 displayed on the solid-state imaging element 42.

[0454] Figure 18C It is shown Figure 18B The image shown is a diagram of the display area 1402. Here, the display area 1402 is the cut-out display area described above.

[0455] Figure 18D This is a diagram showing the cut-out imaging area of ​​the image with the imaging area 1402 cut out, and Figure 18E It shows that by performing Figure 18D The image shown is a diagram of the video obtained after distortion correction of the cut-out display area. Cut-out processing is performed during distortion correction of the cut-out display video, therefore... Figure 18E The video shown has a wider field of view than... Figure 18D The viewing angle of the cut-out imaging area shown is small.

[0456] Figure 19 yes Figure 7A The flowchart of the image stabilization subroutine for step S900 is shown below. Reference will be made below. Figure 25A and Figure 25B To illustrate this process, as described above, this process is performed by the display device control unit 801 of the display device 800.

[0457] In step S901, gyroscope data of the current frame and previous frame are obtained from the frame elements of video file 1000, and the deviation V calculated for the previous frame in step S902 is obtained. n-1 Det Then, based on this information, the approximate deviation V is calculated. n Pre In this embodiment, the current frame is the frame currently being processed, and the previous frame is the frame preceding the current frame.

[0458] In step S902, the detailed deviation V is obtained from the video. n Det Deviation is detected by calculating the degree of movement of feature points in the current frame relative to feature points in previous frames.

[0459] For feature point extraction, known methods can be used. For example, a luminance information image obtained by extracting only the luminance information of the frame can be generated. The image obtained by shifting the luminance information image by one to several pixels can be subtracted from the original image, and pixels with absolute values ​​equal to or greater than a threshold can be extracted as feature points. Alternatively, edges extracted by subtracting the image obtained by applying a high-pass filter to the original luminance information image can be extracted as feature points.

[0460] The difference is calculated multiple times while the brightness information images of the current frame and the previous frame are shifted by one to several pixels each time. The position where the difference between the pixels of the feature point is small is also calculated, thereby calculating the amount of shift.

[0461] As will be explained below, multiple feature points are required, so it is preferable to extract feature points by dividing the images of the current frame and previous frames into multiple blocks. Block segmentation can also be based on the number of pixels or aspect ratio of the image, and is typically preferred to have 12 blocks of 4×3 to 96×64 blocks. When the number of blocks is small, it is not possible to correctly correct for the trapezoidal and optical axis rotations caused by the excitation of the imaging unit 40 of the camera body 1. On the other hand, when the number of blocks is too large, the size of each block is small, and the feature points are close to each other, which may contain errors. Therefore, the optimal number of blocks should be appropriately selected based on the number of pixels, the ease of finding feature points, and the viewing angle of the subject.

[0462] In calculating the displacement, the difference needs to be calculated multiple times by shifting the brightness information images of the current frame and the previous frame by one to several pixels each time, thus increasing the computational load. Therefore, in order to calculate the displacement V... n Pre Calculating the number of shifted pixels can significantly reduce the computational cost of the actual shift by only calculating the differences near these pixels.

[0463] Next, in step S903, the detailed deviation V obtained in step S902 is used. n Det Perform image stabilization correction, then process and exit from this subroutine.

[0464] Traditionally, methods used in image stabilization include Euclidean transformation, which can perform rotation and parallel translation; affine transformation, which can perform rotation and parallel translation; and projection transformation, which can further perform trapezoidal correction.

[0465] Although Euclidean transformations can be used to correct for movement and rotation along the X and Y axes, hand shakiness exists in the forward / backward direction and pan / tilt direction, as deviations occur when the imaging unit 40 of the camera body 1 actually captures images. Therefore, in this embodiment, an affine transformation capable of correcting magnification and skew is used for image stabilization correction. When the coordinates (x, y) of a feature point used as a reference are moved to coordinates (x', y'), the affine transformation can be represented by the following Equation 100.

[0466] [Mathematical Expression 1]

[0467]

[0468] The affine coefficients of the 3×3 matrix represented in Equation 100 can be calculated as long as deviations at at least three feature points can be detected. However, image stabilization correction becomes inaccurate when the detected feature points are located at close range or arranged along a straight line, especially at locations far from the feature points and far from the straight line. Therefore, it is preferable to select detected feature points located at long distances and not arranged along a straight line. Thus, when multiple feature points are detected, feature points that are close to each other are omitted, and the remaining feature points are standardized using the least squares method.

[0469] Figure 18F It shows that by performing Figure 18E The image shown is a screenshot of the video obtained after image stabilization correction in step S903, following distortion correction. Because a cutout process is performed during image stabilization correction, therefore... Figure 18F The video shown has a wider field of view than... Figure 18E The video shown has a narrow field of view.

[0470] By performing such image stabilization, a high-quality video with corrected deviations can be obtained.

[0471] As described above, a series of operations performed by the camera body 1 and the display device 800 included in the camera system according to this embodiment have been explained.

[0472] After the power switch 11 is turned on, and the user selects the motion image mode using the camera mode switch 12 and observes only the front without turning his or her face up / down / left / right, firstly, the face direction detection unit 20 detects the observation direction vo (vector information [0°, 0°]). Figure 12A Then, the recording direction / viewpoint determination unit 30 cuts out the ultra-wide-angle video projected onto the solid-state camera element 42. Figure 12A The video showing the target's field of view of 125° ( Figure 11B ).

[0473] Then, for example, when the user begins to observe without operating the camera body 1. Figure 11A When the child (subject A131) is shown, firstly, the facial orientation detection unit 20 detects the observation direction vl (vector information [-42°, -40°]). Figure 11C Then, the recording direction / viewpoint determination unit 30 cuts out the target field of view 125 l video from the ultra-wide-angle video captured by the camera unit 40. Figure 11C ).

[0474] Thus, depending on the viewing direction, optical correction and image stabilization processing for videos cut out in various shapes are performed in steps S800 and S900 in the display device 800. Accordingly, even when the overall control CPU 101 of the camera body 1 has low specifications, even when the cut-out target field of view 125l ( Figure 11C For example, in cases of videos with significant distortion, such as... Figure 11D As shown, video with distortion and shakiness corrected can also be obtained centered on the child (subject A131). In other words, except for turning on the power switch 11 and selecting the mode using the video mode switch 12, the user can obtain video of him or her observation direction without touching the camera body 1.

[0475] Here, the Pri setting mode will be explained. As mentioned above, since the camera body 1 is a small wearable device, no operation switch or setting screen for changing its detailed settings is provided in the camera body 1. For this reason, an external device such as a display device 800 (in this embodiment, the setting screen of the display device 800) is used instead. Figure 13 Make detailed changes to the settings of camera body 1.

[0476] For example, in the same motion picture shooting, consider the scenario where you want to continue shooting at both 90° and 45° angles. In normal motion picture mode, the angle is set to 90°. Therefore, in this case, after shooting in normal motion picture mode, the motion picture shooting ends, and an operation is needed to change the display device 800 to the camera body 1 setting screen and switch the angle to 45°. However, during continuous shooting, this operation on the display device 800 is cumbersome and may result in missing the desired video recording.

[0477] On the other hand, if the Pri setting mode is preset to the mode used for shooting motion images from a 45° angle, after shooting motion images from a 90° angle has ended, the motion image shooting can be immediately changed to a magnified motion image shooting from a 45° angle simply by sliding the camera mode switch 12 to "Pri". In other words, the user does not need to stop the current camera action and perform the aforementioned cumbersome operation.

[0478] Details of the settings in the Pri settings mode may include not only changes in the viewing angle, but also image stabilization levels specified as "strong," "medium," or "off," as well as changes in speech recognition settings not described in this embodiment.

[0479] For example, when a user switches from motion picture mode to Pri setting mode using camera mode switch 12 while continuing to observe a child (subject A131) in a previously captured scenario, the viewing angle setting value ang changes from 90° to 45°. In this case, the recording direction / viewing angle determination unit 30 cuts out the image captured by the ultra-wide-angle video from the camera unit 40... Figure 11E The dashed box shown represents the target field of view 128 l of video.

[0480] Similarly, in Pri setting mode, optical correction and image stabilization processing are performed in steps S800 and S900 in the display device 800. Accordingly, even when the overall control CPU 101 of the camera body 1 has low specifications, such as... Figure 11F As shown, a video with zoomed distortion and jitter centered on the child (subject A131) can also be obtained, with the zoom level corrected. Although an example of changing the viewing angle setting ang from 90° to 45° in motion picture mode is given, this explanation can also be applied similarly to still picture mode. Furthermore, this explanation can also be applied similarly to the case where the viewing angle setting ang is 90° for motion pictures and 45° for still pictures.

[0481] In this way, by simply switching the camera body 1 to the video mode switch 12, the user can obtain a zoomed video that captures the direction of his or her observation.

[0482] In this embodiment, although the case where the face direction detection unit 20 and the imaging unit 40 are integrally formed in the camera body 1 is described, the configuration is not limited to this. The face direction detection unit 20 can be mounted on the user's body other than the head, and the imaging unit 40 can be mounted on the user's body. For example, the imaging / detection unit 10 according to this embodiment can be mounted on the shoulder or abdomen. However, in the case of mounting on the shoulder, when the imaging unit 40 is mounted on the right shoulder, it can be assumed that the subject arranged on the left side is obscured by the head; therefore, a configuration with multiple imaging units mounted on the left shoulder, etc., to supplement the mounting is preferable. Furthermore, in the case of the abdomen, parallax occurs spatially between the imaging unit 40 and the head; therefore, as in Embodiment 3, it is preferable to perform correction / calculation of the observation direction for parallax correction.

[0483] Example 2

[0484] In Example 2, reference will be made Figure 20A , Figure 20B , Figure 21 , Figures 22A to 22F and Figures 23A to 23E This section details the method for calibrating / adjusting individual differences among users wearing the camera body 1.

[0485] Basically, this embodiment will be described as a derivative of Embodiment 1. For this reason, components in the camera system according to Embodiment 2 that are identical to those in the camera system according to Embodiment 1, using the same reference numerals, will be omitted from repeated descriptions, and other components will be described in detail with additional notes as needed.

[0486] Users equipped with the camera body 1 have individual differences and adjustment variations, represented by physique, the tilt and angle of the neck area to which the camera body 1 is attached, clothing status represented by the collar when equipped, and adjustment margins of the straps 82L and 82R. For this reason, the optical axis center of the camera lens 16 of the camera body 1 is typically not the same as the field of view center when the user is facing forward (hereinafter referred to as the user's natural state). Preferably, the user does not set the optical axis center of the camera lens 16 of the camera body 1 as the center of the recording area (target field of view 125) used for cutting out video as is, but rather sets the field of view center during the user's posture or operation as the center of the recording area.

[0487] Furthermore, individual differences exist not only in the center of the field of view in the user's natural state, but also in the center of the field of view when the user tilts their neck in various directions, including vertical and horizontal tilts, and in the movable area of ​​the neck. For this reason, individual differences also exist in the relationship between the facial orientation (observation direction) detected by the facial orientation detection unit 20 and the center position of the target field of view 125 set according to the observation direction (hereinafter referred to as the field of view center position). Therefore, a calibration operation is needed to associate the facial orientation and the field of view center position.

[0488] Generally, calibration is preferred as a preferred method. Figure 7A This is part of the preparation operation process (step S100). Normally, although it is assumed that the calibration operation is performed upon first activation of the camera body 1, it can also be performed after a predetermined time has elapsed since calibration, or when the camera body 1 causes a deviation in the user's position relative to the previous calibration. Furthermore, a calibration operation can be performed if the face orientation detection unit 20 cannot detect the user's face. Additionally, if it is detected that the user has put on or removed the camera body 1, a calibration operation can be performed in a scenario where the user puts on the camera body 1 again. Thus, it is preferable to perform the calibration operation appropriately at the timing deemed necessary for proper use of the camera body 1.

[0489] Figure 20A and Figure 20B This is a diagram detailing the calibrator 850 used in the calibration process according to Embodiment 2. In this embodiment, the case where the calibrator 850 is also used as a display device 800 will be described.

[0490] In addition to including as Figure 7D In addition to the button A802, display unit 803, built-in camera 805, face sensor 806, and angular velocity sensor 807, the display device 800 shown also includes a position determination index 851 and a calibration button 854. In this embodiment, button B804 according to Embodiment 1 is not used, and as will be explained below, button B804 can be replaced by calibration button 854, so it will not be illustrated here.

[0491] Figure 20A This shows the case where the position determination indicator 851 is a specific pattern displayed in the display unit 803, and Figure 20B The diagram shows the appearance of the calibrator 850 used in the position determination index 851. In this case, the position determination index center 852, which will be explained below, is calculated based on the information about the shape of the calibrator 850.

[0492] Location determination index 851 is not limited to Figure 20A and Figure 20B The example shown can be configured separately from the calibrator 850. The position determination indicator 851 can be arbitrary, as long as it satisfies the condition that its size can be easily measured and its shape can be easily positioned facing the user's face. For example, the position determination indicator can be the lens cap of the camera lens 16 or the charging unit of the camera body 1. In any case, the basic concept of calibration operation is the same; therefore, in the following description, it will primarily be used... Figure 20A The calibrator shown is 850.

[0493] Although described in Embodiment 1, the functionality of the calibrator 850 may be included in the display device 800. Furthermore, similar to the display device 800, the calibrator 850 may be a dedicated device, or, for example, a general smartphone or tablet terminal other than a dedicated device.

[0494] The position determination indicator 851 is an indicator displayed in the display unit 803 of the calibrator 850, and is a graphic representation capable of calculating the horizontal width L851a, the vertical width L851b, and the center 852 of the position determination indicator. In the calibration process described below, the user faces towards the vicinity of the center of the position determination indicator 851; therefore, it is preferable that the position determination indicator 851 has a shape that can be easily perceived as the center of the field of view. Figure 20A Although a circular shape with a black circle at the center of the cross is shown, the location indicator is not particularly limited to this shape. In addition, the location indicator can be a rectangular, triangular, or star-shaped graphic, or, for example, a character icon.

[0495] The position determination indicator 851 is captured by the imaging unit 40 of the camera body 1. Based on the captured video, the display device control unit 801 (position calculation unit / distance calculation unit) calculates the distance between the imaging / detection unit 10 and the calibrator 850, as well as the position coordinates of the position determination indicator 851 shown in the image range. In this embodiment, although such calculation is performed by the display device 800 integrated with the calibrator 850, when the calibrator 850 or the position determination indicator 851 is configured separately from the display device 800, such calculation is performed by the overall control CPU 101 on the camera body 1 side.

[0496] The angular velocity sensor 807 can measure the movement of the calibrator 850. Based on the measurement value obtained by the angular velocity sensor 807, the display device control unit 801 calculates movement information representing the position and orientation of the calibrator 850, which will be described below.

[0497] The calibration button 854 is the button that the user presses when facing the center of the position determination indicator 851 during the calibration process described below. Figure 20A In this context, although the calibration button 854 is a touch button displayed in the touch panel type display unit 803, buttons A802 and B804 can be configured to be used as calibration buttons 854.

[0498] Next, refer to Figure 21 The flowchart shown illustrates in detail the calibration process performed when cutting out video from the ultra-wide-angle image captured by the camera unit 40 according to the user's facial orientation and performing image processing on the video, as described in Embodiment 1.

[0499] Figure 21 This is a flowchart of the calibration process according to this embodiment, performed by the camera body 1 (first calibration unit) and the calibrator 850.

[0500] As an aid to explanation, in Figure 21 In this context, the steps for accepting user operations from the camera body 1 and the calibrator 850 are inserted into the box where the operating body is the user. Additionally, in... Figure 21 In this process, the steps performed by the display device control unit 801 of the calibrator 850 upon receiving the user's operation are inserted into the frame where the calibrator 850 is the operating entity. Similarly, in Figure 21 In this context, the steps performed by the overall control CPU 101 of the camera body 1 by receiving the above-mentioned user operations are inserted into the box where the operating subject is the camera body 1.

[0501] More specifically, in Figure 21In steps S3104 and S3108, the camera body 1 is the operating subject, and in steps S3101, S3105, and S3106, the user is the operating subject. Additionally, in steps S3102, S3103, S3107, and S3110, the calibrator 850 is the operating subject.

[0502] When the process begins, firstly, in step S3101, with the calibrator 850 powered off, the user powers on the calibrator 850 via operation button A802. Similarly, with the camera body 1 powered off, the user powers on the camera body 1 by switching the power switch 11 to the ON position. Afterwards, the user establishes a connection between the calibrator 850 and the camera body 1. When this connection is established, the display device control unit 801 and the overall control CPU 101 each enter calibration mode.

[0503] In addition, in step S3101, the user wears the camera body 1, adjusts the length of the straps 82L and 82R and the angle of the camera body 1, and places the camera body 1 in an appropriate position, thereby forming a state in which the camera / detection unit 10 can perform imaging.

[0504] In step S3102, the display device control unit 801 (first display unit) displays the position determination index 851 in the display unit 803.

[0505] Next, in step S3103, the display device control unit 801 indicates to the user the positions (designated positions) to be shaded by the calibrator 850 in the indication display 855. In this embodiment, five positions, including the front position, upper right position, lower right position, upper left position, and lower left position, are indicated sequentially as designated positions. However, the designated positions are not limited to these, as long as calibration can be performed.

[0506] In step S3104, the overall control CPU 101 activates the camera unit 40 to be in a camera-capable state and activates the face orientation detection unit 20 to be in a state that can detect the user's face orientation.

[0507] In step S3105, the user uses calibrator 850 to cover the designated location indicated in step S3103.

[0508] Next, in step S3106, while maintaining the position of the calibrator 850 in the designated position, the user faces his or her face toward the position determination index 851 and places the center of the user's field of view on the position determination index 851.

[0509] In step S3107, the display device control unit 801 (second display unit) notifies the user of the start of calibration at the specified position in the indication display 855 and displays the calibration button 854. When the display device control unit 801 determines that the user has seen the center 852 of the position determination indicator 851 in front of the field of view, the processing in step S3107 is performed.

[0510] In step S3107a, when the user presses the calibration button 854, the display device control unit 801 sends a calibration instruction to the camera body 1.

[0511] In step S3108, the overall control CPU 101 (acquisition / detection unit) acquires an ultra-wide-angle image showing the position determination index 851 as captured by the camera unit 40, based on the calibration instruction from the calibrator 850, and uses the face direction detection unit 20 to detect the face direction. Then, the overall control CPU 101 (generation unit) calculates the position coordinate information of the position determination index center 852 in the acquired ultra-wide-angle image and generates information representing the relationship between the calculated position coordinate information and the detected face direction.

[0512] The following will be referenced Figures 22A to 22F The details of steps S3103 to S3108 will be explained below.

[0513] Figures 22A to 22F This diagram illustrates the calibration operation for the user's frontal orientation. Accordingly, the center position of the field of view in the user's natural state is aligned with the center position of the target field of view 125 of the video perceived by the imaging unit 40 of the camera body 1.

[0514] Figure 22A This demonstrates the calibration operation performed when the user is facing forward. Figure 21 A diagram showing the screen displayed in the display unit 803 of the calibrator 850 during step S3103.

[0515] like Figure 22A As shown, in the display unit 803 of the calibrator 850, a position determination index 851 is displayed and an indication display 855 indicates the position where the user is expected to place the position determination index 851.

[0516] In the indicator display 855, characters are used to write the indication, such that the position determination indicator 851 is positioned at the center of the field of view when the face is facing forward. The indication displayed in the indicator display 855 is not limited to indications using characters, and may be indications using other methods such as diagrams, photographs, or moving images.

[0517] Alternatively, it can display an indicator like 855, as in a typical tutorial, followed by a location determination indicator 851.

[0518] Figure 22B It shows the user based on Figure 22A The indicated display shows a three-dimensional view of the front side of the device covered by the calibrator.

[0519] User based Figure 22A The indicator shown in 855 is used to cover the front side with calibrator 850, and calibrator 850 is positioned such that position determination indicator 851 is at the center of the field of view when the face is facing forward (steps S3105 and S3106). Afterwards, the user presses calibration button 854. Figure 22A The determination in step S3107 is performed based on the pressing of the calibration button 854. The specific sequence of this determination method will be explained below.

[0520] After that, after inspection Figure 22A After the indicator shown at 855 changes to a notification “Calibration in front direction begins!”, the user presses the calibration button 854 (step S3107a).

[0521] Based on the pressing of the calibration button 854, the camera unit 40 acquires the captured video in step S3108.

[0522] Figure 22C It is shown in Figure 22B The diagram shows the entire ultra-wide-angle image perceived by camera lens 16 in the shown state, and Figure 22D This shows the correction Figure 22C This is a schematic diagram of the image obtained by aberrations in an ultra-wide-angle image.

[0523] According to Figure 22B In the indicated state, the user presses the calibration button 854, and in step S3108, the facial orientation detection unit 20 acquires the facial orientation.

[0524] Figure 22E This indicates the user's calibration operation in the forward orientation. Figure 21 This is a schematic diagram of the facial orientation image recorded by the facial orientation detection unit 20 in step S3108.

[0525] As used in Example 1 Figures 8G to 8KThe facial orientation detection unit 20 calculates the angles of the face in the horizontal / vertical directions using the distances and angles between the chin tip positions 207, 207r, and 207u and the neck position 206. However, similar to the image center, the distance and angle values ​​between the chin tip positions 207, 207r, and 207u and the neck position 206 have individual and adjustment differences, as represented by the user's physique, and are not constant. For this reason, in this embodiment, the relationship between the chin tip position and the neck position 206 at the time the calibration button 854 is pressed is defined as the value when the user has a frontal view as the center of the field of view. Accordingly, this relationship can be used as information for accurately calculating the user's facial orientation without relying on individual and adjustment differences.

[0526] Return to reference Figure 21 In step S3109, the overall control CPU 101 determines whether the preparation for calibration in the frontal direction is complete. In other words, it determines whether the information required to calculate the chin apex position 207, neck position 206, and position determination index center 852 has been obtained.

[0527] At this point, if the required information is not obtained, it is determined that the preparation for calibration in the front direction is incomplete ("No" in step S3109), and the operation from step S3102 is repeated so that the missing information in the required information can be obtained again. If the required information is not obtained, obviously, it is not necessary to perform all the operations from step S3102, and only the operation required to obtain the missing information again can be performed.

[0528] Here, the determination in step S3107 is performed using either the face sensor 806 installed in the calibrator 850 or the built-in camera 805. The specific sequence of this determination method will be illustrated using the case where the calibration operation for the user's frontal orientation is performed using the built-in camera 805 as an example. For the case where the face sensor 806 is used, although there are differences between two-dimensional and three-dimensional information, the basic idea is the same, and its explanation will be omitted. However, when the face sensor 806 is used in the determination in step S3107, the face orientation detection unit 20 of the camera body 1 does not perform face detection by projecting infrared rays 823 onto the user during the time period when infrared rays 823 are projected from the face sensor 806 onto the user. This is to prevent interference between infrared rays 23 and 823.

[0529] First, when the user presses in step S3106 Figure 22A When the calibration button 854 is pressed, the display device control unit 801 uses the built-in camera 805 (face detection unit) to take a picture and acquires the built-in camera video 858 showing the user. Figure 22FAdditionally, the display device control unit 801 detects position information from the acquired built-in camera video 858, including the user's front neck 201, chin tip 203, and nose face 204, as well as the camera / detection unit 10 (camera unit 40).

[0530] The determination of whether the user is looking directly at the position determination indicator 851 and the position determination indicator center 852 is made by the display device control unit 801 (determination unit) using the position information detected by the built-in camera video 858.

[0531] As a result of this determination, if it is determined that the user is looking in a different direction, the display device control unit 801 displays a message on the indication display 855 indicating that accurate information cannot be obtained. Based on this, an instruction can be given to the user to perform a recalibration operation.

[0532] In situations where the display device control unit 801 can use the built-in camera video 858 to determine that an accurate calibration operation cannot be performed, such as when the camera / detection unit 10 is tilted by a predetermined amount or more, or when the face orientation detection window 13 is obstructed or dirty, the display device control unit 801 can also display information on the indication display 855 indicating that accurate information acquisition cannot be performed.

[0533] In addition, by using the built-in camera video 858 obtained in step S3107 and the ultra-wide-angle image obtained in step S3108, the information required for parallax correction, which will be described in Example 5, can be obtained.

[0534] More specifically, before the camera unit 40 captures an image of the position determination indicator 851 in step S3108, information about the size of the position determination indicator 851 (horizontal width L851a and vertical width L851b) is sent from the display device 800 to the camera body 1. Accordingly, the overall control CPU 101 can use the size information of the position determination indicator 851 and the video of the position determination indicator 851 shown in the ultra-wide-angle image acquired in step S3108 to calculate the distance between the camera / detection unit 10 and the position determination indicator 851. The position determination indicator 851 is arranged in a calibrator 850, which is housed in the same housing as the built-in camera 805, and... Figure 22B The calibrator 850 is approximately facing the user, so it can be understood that the camera 805 and the imaging / detection unit 10 are at the same distance.

[0535] Similarly, in step S3107a, the built-in camera 805 takes a picture. Figure 22FBefore the built-in camera produces video, information about the size of the camera / detection unit 10 is sent from the camera body 1 to the display device 800. Accordingly, the display device control unit 801 (vertical distance calculation unit) can use the information about the size of the camera / detection unit 10 and... Figure 22F The built-in camera video 858 shows the video from the camera / detection unit 10 to estimate the vertical distance 5070 between the optical axis center of the camera lens 16 and the user's viewpoint position. Additionally, the display device control unit 801 can also estimate the distance 2071 between the camera lens 16 and the user's jaw tip 203. Distance 2071 may be the distance between the facial orientation detection window 13 and the jaw tip 203.

[0536] Here, in order for the facial orientation detection unit 20 to calculate the user's neck position 206 and jaw tip position, according to the design purpose of the facial orientation detection unit 20, the user's face needs to be moved away from the facial orientation detection window 13 by a predetermined distance or more. Therefore, this estimation result can be set as one of the judgment conditions when determining whether the facial orientation detection unit 20 can accurately detect facial orientation.

[0537] Return to reference Figure 21 By using the above method, in step S3109, the overall control CPU 101 can obtain the required information, and if it is determined that the calibration preparation in the positive direction has been completed, the process proceeds to step S3110.

[0538] In step S3110, the display device control unit 801 (first calibration unit) calculates the required information for offsetting the cutout center position, so that individual differences and adjustment differences are absorbed, and offsets the cutout center position based on the information.

[0539] The specific details of the calculations performed in step S3110 are as follows.

[0540] When the user is in an ideal state according to the design values ​​and ideally wearing the camera body 1, in Figure 22C The position of the center 856 of the ultra-wide-angle image obtained in step S3108 is approximately the same as the position of the position determination index center 852 shown in the ultra-wide-angle image. However, in reality, there are individual differences and adjustment differences, such as those represented by the physique of the user mentioned above. Therefore, the positions of the center 856 of the ultra-wide-angle image and the position determination index center 852 are usually not the same.

[0541] For the user, the preferred cutout center position is not the image center represented by the camera body 1 (that is, center 856), but the position of the field of view center in the user's posture and operation (in other words, the position of the position determination index center 852 in the ultra-wide-angle image).

[0542] For this reason, the deviation 857 between the position determination index center 852 and the center 856 in the ultra-wide-angle image is measured, and the cutout center position is shifted to a value based on the position determination index center 852 rather than to the center 856 of the camera body 1. Additionally, a similar method is used to shift the facial orientation detected by the facial orientation detection unit 20 at this time.

[0543] As a specific offset method, such as Figure 22C The deviation 857 is measured for an ultra-wide-angle image. This deviation is divided into a horizontal deviation 857a and a vertical deviation 857b. After appropriate conversion processing based on the method of projecting the entire viewing angle, the offset can be determined.

[0544] In addition, such as Figure 22D As shown, after performing appropriate transformation processing on the ultra-wide-angle image according to the projection method, the offset can be determined. In other words, the deviation 857a between the center 856a of the transformed ultra-wide-angle image and the center 852a of the position determination index is measured. Figure 22D (not shown in the figure), and the offset can be determined by dividing the deviation 857a into a horizontal deviation 857c and a vertical deviation 857d.

[0545] Will adopt Figure 22C and Figure 22D Which of the offset methods shown can be arbitrarily determined taking into account the processing load and purpose of the camera system.

[0546] By performing the aforementioned calibration operation in the frontal direction, regardless of individual differences or adjustment differences, the facial orientation of each user during the equipment process, the field of view center in the facial orientation within the ultra-wide-angle image, and the facial orientation of the facial orientation detection unit 20 can be appropriately correlated with each other.

[0547] To date, although the calibration operation for the front direction (upper right, lower right, upper left, and lower left) has been explained, similar calibration operations are required for the other four directions (upper right, lower right, upper left, and lower left).

[0548] Therefore, in Figure 21 When the processing in step S3110 ends, the processing proceeds to step S3111.

[0549] In step S3111, if it is determined that there is a direction among the five directions (front, upper right, lower right, upper left, and lower left) that has not undergone calibration, the direction to be calibrated is changed to that direction, and the process returns to step S3103. Accordingly, the calibration operation is repeated similarly for the other directions except for the front direction where the calibration operation has been completed.

[0550] Despite Figure 21 It is not shown in the diagram, but in step S3111, if it is determined that there is no direction for which no calibration operation has been performed, the process ends directly.

[0551] Figures 23A to 23E This diagram illustrates the calibration operation for the user's right hand moving upwards (the upper right direction in an ultra-wide-angle image). Figures 23A to 23E respectively with Figures 22A to 22E Correspondingly, and with the same basic operations, common explanations will be omitted.

[0552] Here, as Figure 23A As shown, in the indicator display 855, an indication using characters is written, such that the position determination indicator 851 is arranged to have a field of view centered on the face when facing the upper right side.

[0553] Figure 23B Show user based on Figure 23A The indicated display shows a perspective view of the appearance of the upper right side of the indicator represented in 855, with the calibrator 850 covering it.

[0554] Figure 23C It is shown in Figure 23B The diagram shows the entire ultra-wide-angle image perceived by the camera lens 16 in the shown state.

[0555] like Figure 23C As shown, as a specific offset method, firstly, the deviation 857 between the center 856 of the ultra-wide-angle image and the center 852 of the position determination index is measured. Then, the measured deviation 857 is divided into a radial deviation 857e and an angular deviation 857f, and the offset can be determined after appropriate transformation processing based on the projection method of the entire viewing angle.

[0556] In addition, such as Figure 23D As shown, after performing appropriate transformation processing on the ultra-wide-angle image according to the projection method, the offset can be determined. In other words, the deviation 857a between the center 856a of the transformed ultra-wide-angle image and the center 852a of the position determination index is measured. Figure 23D (not shown in the figure), and the offset can be determined by dividing the deviation 857a into a radial deviation 857g and an angular deviation 857h.

[0557] Reference Figures 22A to 22E In determining the offset, a method is used that divides the deviation into vertical and horizontal directions. Conversely, in the reference... Figures 23A to 23D The determination of the offset uses a method that divides the deviation into radial and angular directions. However, the difference between these methods lies only in the convenience of explanation, and any method can be used.

[0558] In addition, at this time, such as Figure 23E As shown, the facial orientation detection unit 20 obtains the neck position 206 and jaw tip position 207ru required to calculate the facial orientation when the user is facing the upper right direction. For this reason, regardless of individual differences and adjustment differences among users, the facial orientation formed when the user is looking at the direction of the index center 852 (in this case, the upper right direction) can be accurately measured.

[0559] As mentioned above, in Figure 21 The calibration process shown includes calibration not only for the frontal direction but also for the upper right, lower right, upper left, and lower left directions. Therefore, when a user shakes their head in one of the directions (up, down, left, or right), the facial orientation detection unit 20 can accurately measure the direction the user is facing, thus allowing for appropriate use of the camera body 1 regardless of individual or adjustment differences.

[0560] For simplicity, the above description explains how to repeatedly perform calibration operations in the five directions: front, top right, bottom right, top left, and bottom left.

[0561] However, the calibration operation is not limited to this method. For example, a method can be used whereby the user continuously moves the calibrator 850 in a shape such as a letter Z, a spiral, or a polygon according to the instruction display 855, and the position determination index 851 displayed in the calibrator 850 is continuously perceived as the center of the field of view. In this method, while the calibrator 850 is moving according to the method, the display device control unit 801 sends calibration instructions multiple times to the camera body 1. Each time a calibration instruction is received, the overall control CPU 101 acquires the facial direction detected by the facial direction detection unit 20 and the position coordinate information of the center of the position determination index 852 in the ultra-wide-angle image captured by the camera unit 40, and stores both as historical information. Then, the overall control CPU 101 combines the information extracted from the acquired historical information and calculates the relationship between the cutout center position of the video and the user's facial direction. In addition, by using the information acquired on the calibrator 850 side during the movement of the calibrator 850 according to this method, the information extracted from the historical information can be set only as information about the state of the user looking at the position determination index 851. Therefore, for example, by not extracting information about the user's current state of being viewed from historical data, the accuracy of relationship calculations can be improved.

[0562] Additionally, the display device control unit 801 can be configured to send the measurement values ​​acquired by the angular velocity sensor 807 to the camera body 1 when a calibration instruction is given. In this case, the overall control CPU 101 obtains the user's movement method and movement information indicating the position and posture of the calibrator 850 from the measurement values ​​sent from the angular velocity sensor 807, and stores both as historical information. Accordingly, by determining the position coordinates of the index center 852 based on the movement information of the measurement values ​​acquired by the angular velocity sensor 807, the facial direction detected by the facial direction detection unit 20, and the position in the ultra-wide-angle image captured by the camera unit 40, the calibration operation can be performed simply, easily, and accurately.

[0563] However, in this case, the movement information based on the measurement values ​​acquired by the angular velocity sensor 807 and the movement information based on the position coordinate information of the position determination index 851 need to be consistent with each other. For this reason, when using the measurement values ​​acquired by the angular velocity sensor 807, it is necessary to synchronize the communication between the camera body 1 and the calibrator 850.

[0564] As described above, Example 2 illustrates a calibration method that associates the user's facial orientation with the center position of the target field of view 125 in an ultra-wide-angle image, even in the presence of individual and adjustment differences. However, the present invention is not limited to the various forms shown in Example 2, and various modifications and changes can be made within the scope of the concept of the invention.

[0565] Example 3

[0566] In Example 3, reference will be made Figures 24A to 24C , Figure 25A , Figure 25B and Figure 26 This will explain how to prevent video dizziness by using video recordings as a supplementary method.

[0567] Basically, this embodiment will be described as a derivative of Embodiment 1. For this reason, the same reference numerals used for components in the camera system according to Embodiment 3 that are the same as those in the camera system according to Embodiment 1 will be omitted from repeated descriptions, and other components will be described in detail with additional notes as needed.

[0568] Thanks to advancements in video technology, it is now possible to easily enjoy CG and powerful 3D videos that resemble real images.

[0569] On the other hand, in cases where such 3D videos involve dynamic movement, such as VR, or involve significant hand tremors, video motion sickness may easily occur when viewing them. Video motion sickness can cause symptoms such as kinesia, thus increasing concerns about its safety measures.

[0570] When the video is cut out and displayed in the direction the user's face is facing during the recording range display processing (step S500), the video scene is switched at high speed when the user's face moves quickly while the camera unit 40 is recording (step S400).

[0571] Although a user who is rapidly moving his or her face while using the camera unit 40 to record does not have video motion sickness, there is a possibility that a viewer of such a video may have video motion sickness if the video scene includes a completed video that has been assisted in recording in step S1000.

[0572] Furthermore, neither Japanese Patent Application Publication No. 2007-74033 nor Japanese Patent Application Publication No. 2017-60078, which involve videos formed in the direction the user is facing, describe measures to address this type of video dizziness.

[0573] Therefore, in this embodiment, a camera system is provided that controls the camera to prevent video scenes that switch at high speed from being included in the completed video, even when the user moves his or her face quickly while using the camera unit 40 to take a picture, and prevents the viewer from experiencing video dizziness.

[0574] The same reference numerals are used for components in the camera system according to Embodiment 3 that are the same as those in the camera systems according to Embodiments 1 and 2. Repeated descriptions will be omitted, and other components will be described in detail with additional notes as needed.

[0575] For reference Figures 8H to 8K and Figures 10B to 10D As described above, when the camera unit 40 is used to take a picture, the user's face rotates in both the vertical and horizontal directions.

[0576] Therefore, in the following text, the direction and speed of the user's face movement are represented by angular velocity ω, and the distance it moves will be represented by angle θ.

[0577] Angular velocity ω is calculated by dividing the angle θ detected by the facial orientation detection unit 20 by the detection interval.

[0578] Examples of actions involving a person rapidly moving their face include looking back, glancing, and observing moving objects.

[0579] Looking back is an example of forcefully turning around when there is a loud noise behind you.

[0580] A glance is the act of returning to the original position of the face after seeing a potentially interesting change in the field of vision, due to a lack of interest.

[0581] Observing moving objects involves observing birds actively flying in the air and kites in flight.

[0582] When this operation occurs while using the camera unit 40 to record, as described above, when the video is cut out and displayed in the direction the user's face is facing during the recording range display processing, there is a possibility that the viewer of the completed video may experience video dizziness.

[0583] Therefore, if an angular velocity ω equal to or higher than the threshold ω0 is calculated and maintained for a predetermined time or longer (a first predetermined time or longer), the overall control CPU 101 determines that a user has rapidly moved his or her face (one of looking back, glancing, or observing a moving object). Additionally, using the following reference... Figure 25A and Figure 25BThe method to be described determines that if the operation is neither a glance nor an observation of a moving object, the overall control CPU 101 determines that the operation is looking back. In this case, instead of cutting out the video in the direction the user's face is facing as is in the recording range image processing, the overall control CPU 101 performs a delayed cutout, in which the movement of the user's face is cut out with a delay.

[0584] Here, in this embodiment, the threshold ω0 is set to π / 8 rad / s. This is the speed at which a face moves from a 0-degree frontal angle to a 90-degree side angle in four seconds. However, the threshold ω0 is not limited to π / 8 rad / s. For example, based on the frame rate n fps, the threshold ω0 can be set to (n×π) / x rad / s (where x is an arbitrary value), etc.

[0585] As the angular velocity ω, it can be expressed by the following formula, based on the angle θ obtained from the image of frame n. n and the time t to which it is obtained n and the angle θ obtained from the image of the previous frame n-1 n-1 and the time t to which it is obtained n-1 To obtain angular velocity ω n .

[0586] ω n =(θ n -θ n-1 ) / (t n -t n-1 )

[0587] Here, the angular velocity ω can be w n-x to w n The arithmetic mean.

[0588] In addition, in this embodiment, the predetermined time is 0.2 seconds, but it is not limited to this value.

[0589] In the following text, reference will be made to Figures 24A to 24C This is to explain the delayed switchover when the user looks back.

[0590] Although the reference in Example 1 Figures 11A to 11F and Figures 12A to 12G The previous explanation took into account the distortion of the camera lens 16, but in this embodiment, for the sake of simplicity, the distortion of the camera lens 16 will not be considered. Furthermore, the images of the frames are calibrated according to Embodiment 2, and it is assumed in the description that the center of each frame's image coincides with the center of the user's field of view when the image is captured. In addition, to illustrate the case where the face is facing to the side, an example will be given of projecting a light beam with a maximum FOV angle of approximately 192 degrees onto the solid-state imaging element 42.

[0591] Region 4000 represents the pixel area that can be captured by the solid-state camera element 42.

[0592] Image 4001 ( Figure 24A ) and image 4022 ( Figure 24C () is the frame f cut out from the target field of view 125 in the direction the face is currently facing. n The image.

[0593] Image 4002 ( Figure 24A ) and image 4021 ( Figure 24C () is the frame f cut out from the target field of view 125 in the direction the face was facing previously. n-1 The image.

[0594] Distance 4010 ( Figure 24A ) is from frame f n-1 Image 4002 center to frame f n The distance from the center of image 4001. In the following text, from frame f... n-1 The center of the image to frame f n The value of the distance from the center of the image is represented by d.

[0595] Image 4003 ( Figure 24B () is a delayed frame f' cut out from the video projected into region 4000 when the angular velocity ω of the face, based on the facial direction detected by the facial direction detection unit 20, is equal to or higher than the threshold ω0. n The image cut out from the image.

[0596] Distance 4011 is from frame f n-1 Image 4002's center to delayed cut-out frame f' n The distance from the center of image 4003. In the following text, from frame f... n-1 Image center to delayed cut-out frame f' n The value of the distance from the center of the image will be represented by d'.

[0597] The delay distance of 4012 is from frame f n Image 4001 center to frame f' n The distance from the center of image 4003, in the following text, will be represented by d”.

[0598] At this point, when the value d at a distance of 4010 and the value d' at a distance of 4011 are compared with each other, d > d'.

[0599] Next, refer to Figure 24C Here is an example illustrating the method used to determine the value d'.

[0600] Here, we will explain the case where the user rapidly moves his or her face from the front (viewing direction vo(vector information [0°,0°])) to a 90-degree side view in the right direction. In this case, the frame f in which the face is facing the front (viewing direction vo(vector information [0°,0°])) is acquired. n Then, after a short period of time, a frame f is captured showing the face moving from the front to the right at a 90° angle to the side. n+x Image 4022.

[0601] In cases where, to prevent video vertigo, it is necessary to move 90° to the right from the front for t seconds (e.g., 4 seconds) or longer, and the video frame rate is n fps (e.g., 30 fps), d' = (f n+x -f n ) / (n fps×t seconds).

[0602] On the other hand, when from frame f n to frame f' n As the distance "d" increases, the direction the face is facing is not the recording direction, therefore there are instances where the subject the user is seeing is not in frame f. n The possibilities shown in the text.

[0603] Therefore, when the delay time becomes the predetermined time Th delay In the event of a second predetermined time or longer, the cutting is delayed and the cutting is performed in the direction that the face is currently facing.

[0604] Here, the delay time is the time t0 at which the delay begins. Figure 26 The steps shown (S4211) and the current time t during which the face continues to move are shown. n ( Figure 26 The difference between steps S4213 shown.

[0605] In this embodiment, the predetermined value Th delay It is set to 1 second, but is not limited to this. For example, the predetermined value Th can be based on the frame rate n fps. delay Set to 20 / n seconds. At the predetermined value Th... delay With a frame rate of 20 / n seconds, the higher the frame rate, the higher the predetermined value Th. delay The shorter the frame rate, the lower the likelihood of motion sickness, allowing for a shorter delay before returning to the direction the face is currently facing for cutout processing.

[0606] On the other hand, if the delayed cutout stops and a return cutout occurs in the direction the face is currently facing, the video scene is abruptly switched. Users don't naturally perceive this abrupt scene change, so fade-out, fade-in, or any other video effect can be used.

[0607] Additionally, storing the facial trajectory allows for restarting the cutting out of the face in the direction it is currently facing.

[0608] Here, we will refer to Figure 25A This example illustrates how facial trajectories are stored when it is determined that a user is glancing at a screen.

[0609] As described above, the process of stopping the delayed cutout and returning to the direction the user is currently facing not only changes the delay time to the predetermined value Th delay This can be done over a longer period of time, and also while glancing (in other words, when the face is turned back to its original direction immediately after being looked in a particular direction).

[0610] Figure 25A This is a diagram illustrating an example of the trajectory of a user's face during a glance.

[0611] As frame f n-3 The center position of the image 4101 coincides with the center of the user's field of view at the start of facial movement. Afterwards, the user's field of view center moves to the position of frame f. n-2 f n-1 and f n The positions of the center of the image are 4102, 4103, and 4104. In the following text, this movement of the user's field of view center will be referred to as the facial movement vector.

[0612] Afterwards, the facial movement vector pauses temporarily at position 4104, then moves to frame f. nx+1 f nx+2 and f nx+3 The center positions of the image are 4105, 4106, and 4107, and in frame f nx+3 The image stops at position 4107, which is the center of the image.

[0613] In other words, the movement vectors of the face at positions 4101 to 4104 are opposite to the movement vectors of the face at positions 4104 to 4107.

[0614] When detected, such as Figure 25A In the case of a frame group in which the facial movement vectors are consistent with each other in opposite directions, the overall control CPU 101 determines that the frame group is a frame group during a glance.

[0615] In this case, the overall control CPU 101 performs a delayed cutout from the position 4101 where the face begins to move until the position 4104 where the face's movement vector begins to move in the opposite direction. This is because it is assumed that position 4104 is the position showing the subject that the user wants to glimpse.

[0616] On the other hand, after the delayed cutout has started up to position 4104, a return cutout is made in the direction the user is currently facing (in other words, a cutout at position 4107 where the movement of the face has stopped).

[0617] Furthermore, when the user's face is moving, the overall control CPU 101 detects an object near the center of the face, and if the detected object remains in the center of the face, it determines that the user is observing a moving object. In this case, no delayed cutout is performed in this embodiment.

[0618] Figure 25B This is an example diagram showing images of each frame in the context of a user observing a moving object.

[0619] As frame f n-1 The center of image 4121 is located at the center of the user's field of view when the face begins to move. Afterwards, the user's field of view shifts to the center of frame f. n f n+1 f n+2 f n+3 and f n+4 The center position of images 4122 to 4126.

[0620] Birds, as the same subject, are continuously present near the center of images 4121 to 4126 in each frame.

[0621] When detected, such as Figure 25B In the case shown where there are consecutive frames of the same subject near the center of the image, the overall control CPU 101 determines that the frame group is a frame group during the observation of a moving object.

[0622] In this situation, the overall control CPU 101 does not perform a delayed cutout. The reason is that if a delayed cutout is performed while observing a moving object, there is a high probability that the subject will not be shown in the video.

[0623] Furthermore, when a viewer watches a video captured by cutting out images 4121 to 4126 based on the rapid movement of the user's face while observing a moving object, there is a possibility of causing video vertigo. Therefore, the overall control CPU 101 does not cut out images for frame groups during the observation of a moving object, and records an image of the entire pixel area (that is, area 4000) that can be captured by the solid-state imaging element 42.

[0624] Threshold ω0, predetermined time and predetermined value Th delay Each can have a width known as a blind spot.

[0625] Next, we will use Figure 26 The flowchart shown illustrates the video motion sickness prevention process according to this embodiment. This process is performed each time a frame is captured during motion image capture using the camera unit 40, as described in step S400.

[0626] In step S4201, the overall control CPU 101 acquires the facial direction (viewing direction) recorded in the main memory 103 during the facial direction detection processing performed for this frame capture.

[0627] In step S4202, the overall control CPU 101 obtains the position and size (cutout range) of the video recording frame recorded in the main memory 103 during the recording direction / range determination process for this frame capture.

[0628] In step S4203, the overall control CPU 101 (arithmetic operation unit) calculates the angular velocity ω of the face based on the facial orientation during the current frame capture obtained in step S4201, the facial orientation during the previous frame capture stored in the main memory 103, and the frame rate. Then, the overall control CPU 101 determines whether the face has begun to move at an angular velocity ω equal to or greater than a threshold ω0. More specifically, if the user's face moves at an angular velocity ω equal to or greater than the threshold ω0 for a predetermined time (0.2 seconds) or longer, it is determined that the face has begun to move at an angular velocity ω equal to or greater than the threshold ω0. If it is determined that the face has begun to move, the process proceeds to step S4204; if it is determined that the face has not begun to move, the process returns to step S4201. In other words, even if the user's face has moved at an angular velocity ω equal to or greater than the threshold ω0, if the time is less than a predetermined time (less than a predetermined first time), the process returns to step S4201. In addition, if the facial orientation from the previous frame capture is not stored in the main memory 103 and the angular velocity of the face cannot be calculated in step S4203, the process returns to step S4201.

[0629] In step S4204, the overall control CPU 101 determines whether the face has moved by a predetermined angle or more based on the angular velocity ω of the face calculated in step S4203. If the face is determined to have moved, the process proceeds to step S4206; otherwise, the process proceeds to step S4205. In step S4204, it can be determined whether the face has moved for a predetermined time (0.2 seconds) or longer at a predetermined angular velocity or greater ("yes" in step S4203).

[0630] In step S4205, the overall control CPU 101 determines whether the facial movement has stopped based on the angular velocity ω of the face calculated in step S4203. If the face is determined to have stopped, the process returns to step S4201; otherwise, if the face is determined not to have stopped, the process returns to step S4204.

[0631] In step S4206, the overall control CPU 101 determines whether the subject being filmed is moving (whether the user is observing a moving object). If it is determined that the subject is moving, the process proceeds to step S4207; if it is determined that the subject is not moving, the process proceeds to step S4208.

[0632] In step S4207, the overall control CPU 101 does not perform cropping image processing in the recording range image processing of this frame and determines to perform image processing on the original data of the entire area obtained from the entire solid-state imaging element 42, and the processing proceeds to step S4205.

[0633] In step S4208, the overall control CPU 101 stores the facial direction of the current frame captured in step S4201 as the facial direction of the previous frame captured in the main memory 103, and then proceeds to step S4209.

[0634] In step S4209, the overall control CPU 101 (delay unit) determines, in the recording range display processing of this frame, to perform cropping display processing (delayed cropping) on ​​the cutout range centered at a position offset by a distance d relative to the facial direction of the previous frame. Afterwards, the processing proceeds to step S4210.

[0635] In step S4210, the overall control CPU 101 determines whether the start time t0 of the delay time stored in the main memory 103 has been cleared. If it is determined that the delay time has been cleared, the process proceeds to step S4211; if it is determined that the delay time has not been cleared, the process proceeds to step S4212.

[0636] In step S4211, the overall control CPU 101 stores the current time as the start time t0 in the main memory 103 and proceeds to step S4212.

[0637] In step S4212, the overall control CPU 101 determines whether the facial movement has exceeded a predetermined value Th within the delay time based on the angular velocity ω of the face calculated in step S4203. delayThe previous stage is stopped. If it is determined that the movement has stopped, the process proceeds to step S4215; if it is determined that the movement has not stopped, the process proceeds to step S4213.

[0638] In step S4213, the overall control CPU 101 uses the current time as time t. n It is stored in main memory 103 and processed to step S4214.

[0639] In step S4214, the overall control CPU 101 calculates the time t stored in the main memory 103. n The delay time is calculated as the difference between the start time t0 and the start time t0, and it is determined whether the delay time is equal to or longer than the predetermined time Th. delay When the delay time is equal to or longer than the scheduled time Th delay If the condition is met, proceed to step S4215; otherwise, proceed to step S4206.

[0640] In step S4215, the overall control CPU 101 clears the start time t0 stored in the main memory 103 and proceeds to step S4216.

[0641] In step S4216, the overall control CPU 101 determines the recording direction and viewing angle using the recording direction / viewpoint determination unit 30 based on the facial direction detected by the facial direction detection unit 20, and then proceeds to step S4217.

[0642] In step S4217, the overall control CPU 101 records a flag indicating the metadata of the frame and proceeds to step S4201. The flag indicating the metadata attached here is used as a timing for applying video effects (fade-in and fade-out effects) such as those described above during the auxiliary recording as described in step S1000 according to embodiment 1.

[0643] As described above, in this embodiment, when the angular velocity ω of the face becomes a threshold ω0 or greater, the frame facing the face is not cut out as is, but rather the frame facing the face is cut out according to the movement of the face, thus having the effect of reducing video vertigo.

[0644] Example 4

[0645] In Example 4, reference will be made Figures 27A to 27F , Figure 28A and Figure 28B This explains a method for correcting the cutout range of a video based on the speed at which the user's face moves.

[0646] Basically, this embodiment will be described as a derivative of Embodiment 1. For this reason, the same reference numerals used for components of the camera system according to Embodiment 4 that are the same as those used for components of the camera system according to Embodiment 1 will be omitted from repeated descriptions, and other components will be described in detail with additional notes as needed.

[0647] First, the operation of changing the direction of observation will be explained. Typically, when a person finds something of interest in a field of view that is off-center from the center of the field of view and turns their gaze toward that side, first, his or her face moves, and when the amount of movement becomes certain or greater, the body follows the face in a way that exists with a delay.

[0648] In other words, in this case, the camera / detection unit 10 located in front of the clavicle ( Figure 10A The orientation of the camera lens 16 on the camera body 1 does not move during the initial movement when only the face changes direction. Subsequently, when the user changes the orientation of their entire body, the orientation of the camera lens 16 on the camera body 1 also moves. The following explanation will be based on this characteristic of human behavior.

[0649] Furthermore, when the facial orientation detection unit 20 detects the facial orientation, a deviation occurs due to detection error. When the cut-out position of the video is calculated based on the detection result of the facial orientation, which includes this deviation, deviations such as hand shakiness in the motion image assisted in step S1000 occur, resulting in a deterioration in appearance. Therefore, in order to correct the detection of small shakiness, a low-pass filter is applied to the facial orientation detection result to eliminate the small deviation.

[0650] Furthermore, when facial orientation is also detected by following instantaneous facial movements (such as left / right checks while walking on a public road), the motion image assisted in step S1000 becomes a motion image that may easily cause video dizziness. For this reason, in this embodiment, a process is performed to eliminate (smooth) the micro-motion components of the facial orientation detected by following instantaneous facial movements for approximately 1 to 2 seconds. Accordingly, the motion image assisted in step S1000 can be formed into a motion image with a good appearance.

[0651] Next, refer to Figures 27A to 27F This section provides an overview of the cutout range correction process according to this embodiment.

[0652] exist Figures 27A to 27F In the graph shown, the horizontal axis represents time, and the vertical axis represents the actual observation center (…). Figure 27A ), facial direction ( Figure 27B and Figure 27C ), the direction of camera lens 16 ( Figure 27D ), and the cutting position ( Figure 27E and Figure 27F The angle of the vertical axis. In the vertical axis, the upward direction represents the rightward direction.

[0653] Figure 27A It is a graph representing the movement of the actual observation center. Figure 27A The angle of the vertical axis does not represent the angle of the facial direction detected by the facial direction detection unit 20, but rather the user's facial position relative to a fixed position such as the ground (with the ground as a reference). In other words, Figure 27A The curve diagram shown illustrates this: First, the user faces forward, and then begins to face right from about one second later.

[0654] Figure 27B This is a graph representing the detection results (viewing direction vi) acquired by the facial orientation detection unit 20. Figure 27B The reason why the lines in the detection results shown are not smooth is that, as mentioned above, there is a deviation in the detection results due to detection errors. Therefore, in this embodiment, a low-pass filter is applied to the detection results obtained by the face direction detection unit 20.

[0655] Despite Figure 27B It was not detected in the image, but a process was performed to eliminate (smooth out) the facial orientation detected by following the instantaneous movement of the face.

[0656] Figure 27C This shows the effect of a low-pass filter on... Figure 27B The graph shown illustrates the smoothing of the detection results obtained by the face orientation detection unit 20. (See figure.) Figure 27C As shown, it indicates Figure 27B The detection result line shown is smoothed by passing through a low-pass filter. However, due to such a filter, in Figure 27C In the middle, the movement from the detected facial direction from the front to the right begins at approximately 2 seconds, therefore relative to the related movement and Figure 27A The movements shown started almost simultaneously. Figure 27B A delay (time lag) occurs. Figure 27B and Figure 27C The angle of the vertical axis shown is relative to the angle between the vertical axis and the vertical axis. Figure 27A The angle of the camera lens 16 (based on the camera body 1) is different from the ground reference.

[0657] In addition, Figure 27B In, with Figure 27A In contrast, the tilt levels out after about 4 seconds. This means that, as Figure 27DAs shown, since the camera body 1 (in the direction of the camera lens 16) starts to move with the user's body from about 4 seconds, the movement speed of the face direction detected by the face direction detection unit 20 is relatively reduced.

[0658] Therefore, traditionally, the amount of movement of the camera body is calculated. Figure 27D ) and the face direction detection results after smoothing with a low-pass filter ( Figure 27C The position obtained by adding them together is used as Figure 27E The cut-out position is shown (that is, the observation direction as the center of the target field of view 125). However, when the cut-out position is calculated using conventional methods, the cut-out position does not follow the movement of the actual observation center, and the video completed by auxiliary recording becomes a panning video that suddenly accelerates from about 4.5 seconds after the body begins to move.

[0659] In other words, in order to eliminate the incongruity of moving the actual observation center, such as Figure 27F As shown, the cut-out position (desired value) is preferably calculated so that the panning is approximately constant.

[0660] Therefore, in this embodiment, the cutting position is calculated to prevent it from being as follows: Figure 27E That's when you see the panning suddenly accelerate. In such a way... Figures 27A to 27F In the case where there are two types of moving speeds, including cut-out positions of 0° / second and 10° / second, the time delay before the aforementioned time delay (one second before in this embodiment) is used. Figure 27D The amount of movement of the camera body 1 shown is related to Figure 27C The facial orientation detection results shown are added together to calculate... Figure 27F The expected value is shown. However, in reality, although the movement speed of the cut-out position is not limited to the two types mentioned above (in other words, sudden acceleration and sudden stop in the observation direction) and deceleration is applied slowly, the above calculation method does not plot a deceleration curve for the slowly changing expected value. Therefore, in this embodiment, when the movement of the camera body 1 stops, the expected value is plotted as a deceleration curve by assigning the movement speed from the start of the movement in the observation direction to the present or the movement speed of the cut-out position over a predetermined time period in the past to several frames.

[0661] In the following text, reference will be made to Figure 28A and Figure 28B The flowchart shown illustrates the cutout range correction process according to this embodiment.

[0662] In the following description, the parts common to the already described embodiments 1 to 3 will be simplified or omitted.

[0663] Figure 28A yes Figure 7AThe flowchart shown is a subroutine of the recording direction / range determination process according to step S300 of this embodiment.

[0664] In step S4000a, a low-pass filter (smoothing unit) is used to smooth the observation direction vi obtained in the facial direction detection processing of step S200. (See above reference...) Figure 27B The reason for this is that there is a slight deviation in the observation direction vi due to a slight detection error. As a low-pass filter, a simple method using a sampled moving average of the past few observation directions (e.g., five to ten observation directions) can be used. However, in this case, as the number of averaging operations increases, the tracking of facial orientation changes is delayed. Furthermore, if the user immediately changes their orientation to the left after facing right, there is a problem that the observation direction vi when the user is facing the far right cannot be detected.

[0665] Furthermore, the mixing state of detection errors varies depending on the detection method, so it is preferable to appropriately modify the method according to the detection method. Additionally, the method of applying a low-pass filter can be configured to have different effects on the vertical and horizontal directions.

[0666] Furthermore, as mentioned above, in many cases, instantaneous facial movement is not suitable for capturing the user experience as the purpose of this invention for video recording, and for example, there are situations where facial movement is unavoidable in order to check the safety of the left and right sides during the aforementioned walking. Therefore, in this embodiment, the observation direction vi acquired in step S4000a when the face returns to its original direction within approximately two seconds is also smoothed.

[0667] Furthermore, while further security checks are needed in the horizontal and downward directions, no additional security checks are required in the upward direction; therefore, the low-pass filter can be configured not to be applied in the upward direction.

[0668] When using the processing steps S301 to S304 ( Figure 7D Once the cutout range is determined, the process proceeds to step S4000, and the overall control CPU 101 (second calibration unit) performs cutout range correction processing.

[0669] Then, in step S305, after recording the corrected cut-out range, the process exits from the subroutine. (Refer to...) Figure 28B The flowchart shown illustrates the cutout range correction process.

[0670] Figure 28B This is a flowchart of the cutout range correction process in step S4000.

[0671] exist Figure 28BIn step S4001, the overall control CPU 101 (movement speed calculation unit) obtains gyroscope information from the angular velocity sensor 107 (that is, the movement of the camera body 1 in this frame (gyroscope movement amount)).

[0672] In this embodiment, although an angular velocity sensor 107 is used, the sensor is not limited to this; any sensor capable of detecting movement of the camera body 1 is acceptable. For example, a magnetic sensor (not shown) for measuring the magnitude / direction of a magnetic field can be used, or an accelerometer sensor 108 for detecting acceleration can be used. Alternatively, the movement of the camera body 1 can be calculated by extracting feature points and detecting the movement vector by calculating the degree of movement of the feature points. Known methods can be used for feature point extraction. For example, the movement can be calculated by applying a bandpass filter to the image obtained by extracting brightness information from only two images to extract edges, subtracting multiple edge images in the shifted state, and calculating the position of the smallest difference. According to this method, although the computational load increases, hardware such as the angular velocity sensor 107 is not required, thus enabling a lightweight camera body 1, which is one of the preferred embodiments.

[0673] The following text will use the case of obtaining gyroscope information from angular velocity sensor 107 as an example to continue the explanation.

[0674] In step S4002, the moving speed (gyroscope moving speed) of the camera body 1 is calculated based on the gyroscope information obtained in step S4001 and the previously obtained gyroscope information.

[0675] In step S4003, it is determined whether the gyroscope's moving speed calculated in step S4002 is decelerating. If the moving speed is not decelerating ("No" in step S4003), the process proceeds to step S4004. Otherwise, the process proceeds to step S4006.

[0676] In step S4004, the overall control CPU 101 (second calibration unit / observation direction correction unit) calculates the moving speed of the cut-out position based on the cut-out position determined in step S304 and the previously acquired cut-out position. Next, a subtraction is obtained by subtracting the gyroscope moving speed acquired before the time delay caused by applying a low-pass filter from the calculated moving speed of the cut-out position.

[0677] In step S4005, the moving speed and subtraction amount of the cut-out position obtained in step S4004 are stored in the main memory 103, and the process exits from the subroutine.

[0678] In step S4006, the sum of the subtractions stored in main memory 103 is assigned, and an expected value is calculated to make the change in the movement speed of each cutout position stored in the same main memory 103 over a predetermined past time period constant, and the process exits from this subroutine. The predetermined past time period can be the time period from when the cutout position actually starts moving until now, or the time period from when the angular velocity sensor 107 detects the movement of the camera body 1 until now. Alternatively, to simplify the process, the predetermined time period can be set to approximately 0.5 seconds to 3 seconds. The expected value prior to the predetermined past time period is set as the movement speed of the cutout position obtained in step S4004.

[0679] Table 1 below shows the conditions from 27A to Figure 27F The displacement (velocity) of the data is formed into a curve. In other words, Table 1(c) shows the moving velocity of the cut-out position determined in step S304, and Table 1(d) shows the gyroscope moving velocity calculated in step S4002. In addition, Table 1(e) shows the expected value calculated in step S4006.

[0680] [Table 1]

[0681]

[0682] As shown in Table 1, as an example, the scenario will be illustrated where the user first stops in front and slowly looks to the right. Figure 28B The subroutine shown is for the cut-out range correction process.

[0683] First, the user is looking directly at the front, so the gyroscope movement speed calculated in step S4002 is almost 0° / second. In other words, in step S4003, it is determined that the gyroscope movement speed is not decelerating, and the process proceeds to step S4004. In this case, the position of the face does not change, so the movement speed of the cutout position is also 0° / second. Furthermore, the subtraction calculated in step S4004 is also 0° / second.

[0684] Although the user begins to face right approximately one second from now, due to the time delay caused by the low-pass filter, as... Figure 27C As shown, the moving speed of the cutting position remains 0° / second. On the other hand, as... Figure 27D As shown, the camera body 1 remains stationary; in other words, the gyroscope's movement speed is almost 0° / second. Therefore, similar to the time the user remains stationary in front, the subtraction calculated in step S4004 is also 0° / second.

[0685] When the user turns further to the right for approximately 2 seconds, such as Figure 27CAs shown, the moving speed of the cutting position becomes 10° / second. On the other hand, as... Figure 27D As shown, the camera body 1 remains stationary; in other words, the gyroscope's movement speed is still almost 0° / second. Therefore, the subtraction calculated in step S4004 is 10° / second.

[0686] As the user faces further to the right and remains so for approximately four seconds, the user's body also begins to face to the right. In other words, as... Figure 27D As shown, because the orientation of camera body 1 is changing, the gyroscope's movement speed becomes 10° / second. When the body begins to rotate, as... Figure 27B As shown, although the actual angular velocity of the face decelerates the relative velocity between the camera body 1 and the face direction, due to the time delay according to the low-pass filter, Figure 27C The moving speed of the cut-out position shown is still 10° / second at this time point. Therefore, taking into account this time delay, the subtraction calculated in step S4004 becomes 10° / second.

[0687] The user is facing further to the right, and although the gyroscope moves at a constant speed of 10° / second from about 5 seconds onwards ( Figure 27D ),but Figure 27C The moving speed at the cut-out position shown decelerates and becomes 0° / second. Therefore, the subtraction calculated in step S4004 becomes -10° / second.

[0688] Despite Figures 27A to 27F Not shown, but when the user finishes facing right at approximately 6 seconds, the gyroscope's movement speed becomes 0° / second, and in this case, the process enters step S4006 for the first time. In this case, the total of the subtractions calculated up to this point and stored in main memory 103 becomes +10° / second. This total of the subtractions is assigned so that the change in movement speed of each cut-out position stored in main memory 103 over a predetermined time period is constant, and the expected value is calculated. Here, Figure 27C The moving speeds of the cut-out positions shown are 10° / s, 10° / s, 10° / s, and 0° / s up to the present time period (2 to 6 seconds) as shown in Table 1. Therefore, to keep the change in moving speed of each cut-out position constant (here, no change) during this time period, all expected values ​​for the 2 to 6 second time period are set to 10° / s.

[0689] In this embodiment, for the sake of simplicity, although an interval of one second is given, the frame rate for capturing motion images is typically 24 to 60 fps. On the other hand, in many cases, it is not necessary to perform face orientation detection and gyroscope detection 60 times per second; therefore, it is preferable to change the timing of face orientation detection processing and cutout range correction processing from the camera timing. For example, although camera recording is performed at 60 fps, the timing of face orientation detection processing and cutout range correction processing can be set to 10 fps, and can be appropriately changed considering factors such as application and power consumption.

[0690] In this embodiment as described above, the following example is illustrated: when the viewing direction moves significantly, at the point in time when the movement of the face and the movement of the body (camera body) are combined, the amount of movement of the viewing direction can be kept constant, so that a motion image with poor appearance is not formed due to the change in the speed of the line of sight movement.

[0691] In this embodiment, although an example of cropping an ultra-wide-angle image based on the viewing direction is illustrated, the configuration is not limited to this. For example, the overall control CPU 101 (camera orientation changing unit) can be configured to change the camera orientation of the camera unit 40 based on the viewing direction. However, in this case, it is necessary to arrange a mechanism (drive unit) in the camera body 1 that mechanically drives the camera orientation of the camera unit 40 (more specifically, the orientation of the camera lens 16 and the solid-state imaging element 42 in the yaw and pitch directions) using a method not shown in the figure.

[0692] Furthermore, although the smoothing of the face direction detection result is exemplified as an example in this embodiment, similar processing for delaying the following of the face direction is preferably performed when the overall control CPU 101 (image stabilization unit) performs the image stabilization control described in Embodiment 1.

[0693] Example 5

[0694] In Example 5, reference will be made Figure 29A , Figure 29B , Figure 30 , Figure 31 , Figure 32A , Figure 32B , Figure 33A , Figure 33B and Figures 34A to 34C This describes a method for reducing the difference between the user's field of view and the assisted recorded video (hereinafter referred to as "recorded video") caused by parallax deviation due to the positional difference between the user's eye position and the equipment position of the camera / detection unit 10.

[0695] Basically, this embodiment will be described as a derivative of Embodiment 1. For this reason, components in the camera system according to Embodiment 5 that are the same as those in the camera system according to Embodiment 1, which use the same reference numerals, will be omitted from repeated descriptions, and other components will be described in detail with additional notes as needed.

[0696] First, for ease of understanding, the difference between the user's field of view and the recorded video that occurred in Example 1 will be explained.

[0697] Figure 29A and Figure 29B This is a schematic diagram illustrating the relationship between the user 5010's field of view and the target field of view when the subject located at close range is set as the observation object 5020 in Embodiment 1.

[0698] Figure 29A This is a schematic diagram showing a video 5900 including the object of observation 5020 shown in the solid-state camera element 42, and Figure 29B This is a schematic diagram showing the positional relationship between user 5010 and observed object 5020.

[0699] like Figure 29B As shown, with the object 5020 located below the user's eye level 5011, the user's face direction 5015 is facing downwards. At this time, the object 5020 is shown against a background such as a floor (not shown in the figure) in the user's field of view.

[0700] In the case of Embodiment 1, the observation direction 5040 is parallel to the facial direction 5015 of the user detected by the facial direction detection unit 20. Figure 29B Set the recording direction. For this reason, such as... Figure 29B As shown, when the subject at close range is the observation object 5020, there is a problem that the area excluding the observation object 5020 is set as the target field of view 5045.

[0701] In this case, a background (ceiling, etc., not shown in the figure) different from the background shown in the user 5010's field of view is formed, and the recording direction needs to be set to the direction 5030 of the target field of view 5035, which includes the observed object 5020, instead of the observation direction 5040.

[0702] The aforementioned problem is caused by parallax deviation, which arises from the positional difference between the position of the user 5010's eyes 5011 and the equipment position of the camera / detection unit 10. Therefore, in this embodiment, parallax correction processing is performed, in which the recording direction set based on the user 5010's facial orientation is appropriately adjusted according to the parallax deviation.

[0703] Figure 31 This is a block diagram showing the hardware structure of the camera body 1 according to this embodiment.

[0704] According to the hardware structure of the camera body 1 in this embodiment and Figure 5 The only difference in the hardware structure of the camera body 1 according to Embodiment 1 is the presence of a distance measurement sensor 5100. Although the placement of the distance measurement sensor 5100 within the camera body 1 is not specifically defined, in this embodiment, as... Figure 30 As shown, the distance measurement sensor 5100 is arranged in the outer frame of the stop switch 15.

[0705] The distance measurement sensor 5100 is a sensor that measures the distance to an object. The structure of the distance measurement sensor 5100 is not particularly limited. For example, the distance measurement sensor 5100 can be an active sensor that projects infrared light, a laser beam, or millimeter waves onto an object and measures the distance to the object based on its reflection. Alternatively, the distance measurement sensor 5100 can be a passive sensor that measures the distance to an object based on the phase difference of the light beam transmitted through the camera lens 16.

[0706] The distance measurement sensor 5100 is connected to and controlled by the overall control CPU 101.

[0707] Figure 32A and Figure 32B This is a schematic diagram illustrating the relationship between the user, the calibrator 850, and the target field of view 5080 during calibration, which includes parallax correction processing according to this embodiment.

[0708] Figure 32A This is a schematic diagram showing a video 5900 including a calibrator 850 shown in a solid-state camera element 42, and Figure 32B This is a schematic diagram showing the positional relationship between user 5010 and calibrator 850.

[0709] Figure 32A The target field of view 5080 shown is the target field of view without calibration including the parallax correction process described below, and the face direction 5015 detected by the face direction detection unit 20 is the frontal direction.

[0710] on the other hand, Figure 32A The target field of view 5090 shown is the target field of view under calibration including the parallax correction process to be described below, and the face direction 5015 detected by the face direction detection unit 20 is the frontal direction.

[0711] Figure 33A Is as Figure 7A The flowchart shown is a part of the preparation process in step S100 according to this embodiment, specifically the parallax correction mode processing. Reference will also be made below. Figure 32A and Figure 32B To explain the details of this process.

[0712] exist Figure 7A In the preparation operation process according to step S100 of the process shown, when the parallax correction mode is activated according to the operation of the calibrator 850 by the user 5010 (step S5101), the display device control unit 801 displays the position determination index 851 (step S5102).

[0713] Subsequently, the display device control unit 801 determines the position (specified position) of the mask calibrator 850 based on the user's instruction. More specifically, this is achieved by performing actions such as... Figure 22A The indicated display 855 shows that the display device control unit 801 instructs the user 5010 to determine the indicator 851 at the front cover position at eye level (step S5103).

[0714] When user 5010 gives the instruction display 855, user 5010 places the mask calibrator 850 at the designated position indicated in step S5103, and aligns the face direction 5015 towards the position determination indicator 851 (front side). At this time, user 5010, position determination indicator 851, and camera / detection unit 10 have the following characteristics: Figure 32B The positional relationship is shown.

[0715] Subsequently, if it is determined that the user is looking at the position determination indicator center 852 directly in front of the field of view, the display device control unit 801 uses the distance measurement sensor 5100 to measure the distance 5050 between the camera / detection unit 10 and the position determination indicator 851. Figure 32B (Step S5104).

[0716] Subsequently, the overall control CPU 101 uses the angular velocity sensor 107 (attitude detection unit) to detect the horizontal axis 5060 of the camera / detection unit 10 (step S5105). Based on this, the video 5900 output in the solid-state imaging element 42 is specified. Figure 32A The horizontal position is 5065.

[0717] Additionally, in step S5105, the overall control CPU 101 acquires the distance 5855 between the center of the position determination index 851 on the video 5900 and the horizontal position 5065. Figure 32A Then, the overall control CPU 101 (angle calculation unit) calculates the angle 5055 formed by the horizontal axis 5060 and the direction of the position determination index 851 seen from the camera / detection unit 10. Figure 32BThe calculation is performed using distance 5855 and information relating to the angle of light incident at a point on video 5900 that is imaged in memory (e.g., built-in nonvolatile memory 102).

[0718] Then, the overall control CPU 101 (vertical distance calculation unit) uses the distance 5050 and the angle 5055 calculated in step S5105 to calculate the vertical distance 5070 between the camera / detection unit 10 and the eye 5011 of the user 5010 (step S5106), and processes the exit from the subroutine.

[0719] Here, although a method different from the method according to Embodiment 2 is described for measuring the vertical distance 5070 between the camera / detection unit 10 and the eye 5011 of the user 5010, the method is not limited thereto. For example, the method described in Embodiment 2 can be used to measure the vertical distance 5070 between the camera / detection unit 5010 and the eye 5011 of the user 5010, and the user 5010 can be allowed to directly input the value of the vertical distance 5070.

[0720] The calibration process, including parallax correction, performed according to this embodiment is similar to that in Embodiment 2. Figure 21 The processes shown in steps S3101 to S3111 are basically the same, so their descriptions will be omitted.

[0721] However, in step S3110, in addition to the processing described in Example 2, a process based on... Figure 33A The vertical distance calculated in the parallax correction mode shown is 5070 ( Figure 32B Parallax correction. In other words, calibration is performed so that the user's field of view 5010 and the target's field of view 125 are consistent with each other at infinity.

[0722] Figure 33B It is for reference. Figure 7A The flowchart of the recording direction / range determination subroutine according to S300 of this embodiment is shown below. (Refer to...) Figures 34A to 34C To illustrate this process. Figure 33B In the diagram, the same reference numerals are assigned to... Figure 7D The steps are repeated, and their repeated descriptions will be omitted.

[0723] exist Figure 33B First, the overall control CPU 101 uses the distance measurement sensor 5100 (distance measurement unit) to acquire distance information of the shooting range (camera area) (step S5301).

[0724] Next, the overall control CPU 101 (generation unit) generates a defocus map 5950 based on the distance information obtained in step S5301 (the measurement result obtained by the distance measurement sensor 5100). Figure 34A (Distance map information) (Step S5302).

[0725] Figure 34A The defocus diagram 5950 shown indicates that in the defocusing diagram... Figure 34C The defocus map shown is generated when the observed object 5020 is floating indoors and being photographed. Here, in order to more easily understand the distance information in the defocus map 5950, the defocus map is represented in a hierarchical manner, dividing it into 6 distance regions (1) to (6) starting from the side with the shortest distance relative to the camera / detection unit 10. However, in practice, the defocus map 5950 can be generated without hierarchical division.

[0726] Subsequently, the overall control CPU 101, based on the defocus map 5950, facial direction 5015, and vertical distance 5070, Figure 32B The orientation of the observed object 5020 as seen from the camera / detection unit 10 is calculated (step S5303). In other words, parallax correction is performed for the observation orientation set based on the facial orientation.

[0727] After that, proceed Figure 7D The processes shown in S301 to S305 are processed, and the process exits from the subroutine.

[0728] By using the defocus map 5950 generated in this way and the detection results of the face direction 5015, the orientation of the observed object 5020 as seen from the camera / detection unit 10 can be calculated. Due to the presence of a reference... Figure 29A and Figure 29B The aforementioned parallax bias means that, without generating a defocus map 5950, the distance between the distance measuring sensor 5100 and the observed object 5020 cannot be measured.

[0729] The degree of parallax deviation described in this embodiment varies depending on the distance between the user 5010 and the observed object. In other words, for an observed object at a certain distance relative to the user 5010, the effect of parallax deviation can be ignored. Therefore, similarly, in the recording direction / range determination process according to Embodiment 1, video can be cut out and recorded within the target field of view including the observed object. For example, when the user 5010 observes the observed object 5021 ( Figure 34CIn the case where the observed object 5021 is located at a certain or greater intermediate distance from the user 5010 (5), parallax correction for the recording direction can be omitted in step S5303. This is because the observed object 5021 is also included in the target field of view 5043 (which is set based on the recording direction 5041 (observation direction) estimated by the facial direction 5016 detected by the facial direction detection unit 20). Figure 34B )middle.

[0730] On the other hand, according to this embodiment, the distance range between the user and the object being observed by user 5010 that can enter the target field of view can be extended to a closer side compared to Embodiment 1. For example, assume that user 5010 observes object 5020 located in a close-range region (1) that is short in distance from user 5010. Figure 34A In this case, in Embodiment 1, the observation direction 5040 (recording direction) is estimated based on the facial direction 5015 detected by the facial direction detection unit 20. The observed object 5020 is not included in the target field of view 5042 set according to the observation direction 5040. Figure 34B In ), however, in this embodiment, in Figure 33B In step S5303, parallax correction is performed for the observation direction 5040, and a target field of view 5036 including the observed object 5020 is set in the recording direction after the parallax correction. For this reason, as with the case of the observed object 5020, it is possible to capture images of close-range observed objects at a distance from the user 5010 where the effect of parallax deviation cannot be ignored.

[0731] Furthermore, according to this embodiment, the observed object located in the intermediate distance region (5) can be recorded at a more central position compared to the center of the target field of view. For example, when user 5010 is observing the observed object 5021 located in the intermediate distance region (5) Figure 34A In the same manner as in Embodiment 1, without performing parallax correction on the recording direction 5041, a target field of view 5043 is set with the observed object 5021 located at the upper end. On the other hand, in this embodiment, in... Figure 33B In step S5303, parallax correction is performed for the recording direction 5041, and a recording area 5037 centered on the observed object 5021 is generated based on the recording direction after parallax correction.

[0732] Thus, compared to Example 1, when performing parallax correction according to this embodiment, the observed object can be perceived at the center of the cut-out video.

[0733] In this embodiment, parallax correction is also performed for calibration, and after the user's field of view and the recording area are made consistent at infinity, parallax correction is performed. In this parallax correction, the shorter the distance between the user and the observed object during recording, the greater the difference in the recording direction before and after correction. However, parallax correction according to this embodiment can be performed for subjects at finite positions, for example, for subjects at a distance longer than the distance between the calibrator 850 and the user's position in the calibration process according to Embodiment 2, or for subjects at a distance shorter than that position.

[0734] Example 6

[0735] In Example 6, reference will be made Figure 35 , Figure 36A and Figure 36B This explains how to determine the cut-out range when the calculation of the observation direction fails.

[0736] Basically, this embodiment will be described as a derivative of Embodiment 1. For this reason, the same reference numerals used for components of the camera system according to Embodiment 6 that are the same as those used for components of the camera system according to Embodiment 1 will be omitted from repeated descriptions, and other components will be described in detail with additional notes as needed.

[0737] In Example 1, as Figure 7A As shown, in the recording direction / range determination process of step S300, the target field of view is set based on the observation direction calculated according to the facial direction detected by the facial direction detection unit 20 in step S200. However, there are situations where the facial direction detection unit 20 is covered by an obstacle such as a collar or hair, the facial direction detection unit 20 malfunctions, or the facial direction detection unit 20 becomes far away from the user. In these cases, the user's facial direction cannot be obtained. In such cases, it is impossible to capture video of the target field of view that the user wishes to capture.

[0738] In Japanese Patent Application Publication No. 2007-74033, when a second camera fails to detect a user, the system does not store the undetected user in the history of user observation information up to the present, and instead performs user detection again. Furthermore, when filming by following the direction of the face, if face direction detection fails, the system determines the camera direction based on the situation to capture video that does not significantly deviate from the user's intention.

[0739] Conversely, in this embodiment, when the user's facial orientation can be detected, similar to Embodiment 1, the facial orientation detection unit 20 detects the facial orientation and captures a video of the target field of view based on the observation orientation calculated from the facial orientation. On the other hand, when the user's facial orientation cannot be detected and the user's observation orientation cannot be calculated, a video of the target field of view reflecting the user's intention is captured. In other words, in this embodiment, when the facial orientation detection process ends in step S200 and before the recording orientation / range determination process in step S300, the observation orientation determination process is performed. In this process, if the facial orientation detection unit 20 fails to detect the user's facial orientation, the user's intention is determined based on this situation, and the observation orientation is estimated. In other words, a video of the target field of view based on the recording orientation other than the observation orientation calculated from the facial orientation is captured.

[0740] Figure 35 This is a flowchart of the observation direction determination process performed by the overall control CPU 101 according to this embodiment. Referring below... Figure 36A and Figure 36B This will explain the process.

[0741] First, in step S6001, it is determined whether the facial direction detection unit 20 can acquire the facial direction. If the facial direction can be acquired, the process proceeds to step S6004, and the overall control CPU 101 (mode switching unit) switches the processing mode to the facial direction mode (first camera mode), and determines the observation direction calculated based on the facial direction using the method described in Embodiment 1 as the recording direction. After that, the process exits the subroutine.

[0742] On the other hand, if the facial orientation cannot be obtained ("No" in step S6001), the overall control CPU 101 (mode switching unit) causes the processing to proceed to step S6002 to switch to other modes, and determines whether there is a previously tracked subject.

[0743] Here, we will refer to the relationship between the user's viewing direction detection state for each frame and the captured video. Figure 36A Let's explain the judgment and processing in step S6002.

[0744] exist Figure 36A In this context, n is the frame number of the video, θ is the horizontal angle of movement of the user's face, and the user state represents the positional relationship between the user and the observed object in each frame. Furthermore, the entire video represents the ultra-wide-angle video captured by camera unit 40 for each frame, and the captured video represents the images recorded as supplementary data for each frame.

[0745] As shown in the various screens of the user status, Figure 36A The following situation is shown: The user is observing an object represented by "□" located at the bottom of the screen, and the user's observed object cannot be detected for n=5 (that is, for the fifth frame).

[0746] In this embodiment, the current frame is set as a reference, and the four previous frames relative to the current frame are set as a predetermined time period. If the same subject is identified as being present three or more times in the captured video during this predetermined time period, it is determined that there is a subject that has been tracked in the past.

[0747] like Figure 36A As shown, for n = 1 to 4, regardless of the movement angle θ changing by +10° each time, the captured video includes objects marked with "□" that are identified as the same subject. For this reason, for n = 5, it is determined that there is a subject that was previously tracked.

[0748] The judgment criteria for step S6002 can be changed according to the detection cycle of the facial direction and the accuracy of the facial direction detection unit 20.

[0749] refer to Figure 35 If it is determined that there is a subject (the same subject) that was tracked within a predetermined time period in the past (yes in step S6002), the process proceeds to step S6005.

[0750] In step S6005, the processing mode is switched to a pre-subject tracking mode (second camera mode) where the pre-subject orientation is set as the recording direction, and after determining the recording direction to track the subject, the processing proceeds to step S6008. Thus, in this embodiment, even when a previously tracked subject exists despite the inability to detect a facial orientation, the recording direction is determined by switching to the pre-subject tracking mode, thereby reflecting the previous user's intent in the video. Subject recognition and tracking detection technologies within the captured video using the overall control CPU 101 (subject recognition unit) are known, and therefore detailed descriptions will be omitted.

[0751] On the other hand, if it is determined that there is no previously tracked subject ("No" in step S6002), the process proceeds to step S6003.

[0752] In step S6003, it is determined whether a subject pre-registered in the built-in non-volatile memory (subject registration unit) is detected in the latest captured video.

[0753] In this embodiment, the subject is pre-registered by the user specifying an image of the person to be photographed from images stored in the display device 800, the display device control unit 801 identifying the features of the selected person, and sending these features to the overall control CPU 101 within the camera body 1. The subject detected in step S6003 is not limited to this, and may include, for example, subjects included in the captured video obtained at a read completion timer and another detection timer. Furthermore, pattern matching technology is used to determine the consistency between the pre-registered subject and the subject in the latest captured video. Pattern matching technology is known, and therefore its detailed description will be omitted.

[0754] If it is determined that a pre-registered subject has been detected in the latest video (Yes in step S6003), the process proceeds to step S6006.

[0755] In step S6006, the processing mode is switched to the registration subject mode (third camera mode) that sets the direction of the subject detected in step S6003 as the recording direction, and the processing proceeds to step S6008.

[0756] On the other hand, if it is determined that no pre-registered subject is detected in the latest captured video (No in step S6003), it is determined that the subject to be set as the object of observation cannot be estimated, and the process proceeds to step S6007.

[0757] In step S6007, the overall control CPU 101 (viewpoint changing unit) switches the processing mode to a subject loss mode (fourth camera mode). In this subject loss mode, the recording direction before the failure of facial direction detection remains unchanged, and the camera viewing angle is changed to a wide-angle compared to the previous viewing angle. Then, the processing proceeds to step S6008. Furthermore, the recording direction in the subject loss mode can continue to move by the amount of change in the observation direction before the failure of facial direction detection.

[0758] Here, we will refer to Figure 36B To explain the situation in step S6007 where the subject loss mode is formed.

[0759] exist Figure 36B This will be used as an example to illustrate the case where the user's observed object cannot be detected when n=5 (that is, the fifth frame).

[0760] exist Figure 36BIn the examples shown, no significant subjects were found for n=1 to 4, and no pre-registered subjects were found in the video taken at n=5. For this reason, the viewing direction at n=5 was changed by adding inertia to the rightward direction across the entire facial video, which is the direction of movement for n=1 to 4. Additionally, the perspective of cutting out the video from the entire facial video will change to a wide-angle.

[0761] In step S6008, if the recording direction is determined from a direction other than the facial direction in any of steps S6005 to S6007, the overall control CPU 101 (notification unit) notifies the user of an error indicating that the facial direction detection failed (detection error). Afterwards, processing exits from this subroutine. In this embodiment, using... Figure 5 The vibrator 106 shown provides a warning to the user. However, the notification method used in step S6008 is not limited to the method of this embodiment, and any other notification method can be used, such as a warning using LED 17 or a display using a terminal that cooperates with the camera body 1, such as display device 800.

[0762] As described above, in this embodiment, when the facial orientation cannot be detected, the recording direction and viewing angle change according to the situation, thus avoiding the loss of video footage of the user's intended target field of view.

[0763] In other words, in this embodiment, subject tracking is performed when the facial direction cannot be detected but previously tracked subjects or pre-registered subjects can be detected. On the other hand, when such subjects cannot be detected, the viewing angle is changed to a wide angle, which is wider than the defined viewing angle, in order to prevent camera loss and make it easy to re-detect the subject.

[0764] Therefore, it is possible to prevent unwanted videos from being captured due to failure to detect facial orientation.

[0765] Although steps S6001 to S6008 are performed for each frame, even after switching to each mode, the mode can be changed based on mode judgment information indicating whether a facial direction is obtained from the facial direction detection unit 20. For example, in this embodiment, if a pre-registered subject is detected as a result of widening the viewing angle according to the subject loss mode, a switch to the registered subject tracking mode, which sets the direction of the detected subject as the viewing direction, is performed. In this case, the widened viewing angle is returned to the defined viewing angle.

[0766] In addition, in this embodiment, although the mode is changed based on a single judgment, the mode can also be changed based on multiple judgments based on the frame rate and the detection capability of facial direction.

[0767] Example 7

[0768] In Example 7, reference will be made to Figures 37A to 37C and Figures 38 to 40 This explains how to determine the observation direction based on the accuracy (reliability) of facial orientation detection.

[0769] Basically, this embodiment will be described as a derivative of Embodiment 1. For this reason, the same reference numerals used for components of the camera system according to Embodiment 7 that are the same as those used for components of the camera system according to Embodiment 1 will be omitted from repeated descriptions, and other components will be described in detail with additional notes as needed.

[0770] In Example 6, by switching the mode used to determine the observation direction based on whether the facial direction can be detected, recording in a direction unwanted by the user is prevented. On the other hand, as in Japanese Patent Application Publication No. 2007-74033, if the user's facial direction cannot be continuously and stably detected, there is a possibility of recording from an angle unwanted by the user. Here, as an example of not being able to continuously and stably detect the facial direction, such as... Figure 1B As shown, there are cases where the camera body 1's imaging / detection unit 10 is mounted in front of the clavicle, and cases where the accuracy of facial direction detection decreases due to the influence of collars and hair.

[0771] like Figures 37A to 37C As shown, when the user is facing to the right ( Figure 37B and Figure 37C In this case, the area of ​​the jaw and chin hidden by the body and shoulders becomes larger than when the user is facing forward. Figure 37A In other words, the camera body 1 exhibits the following characteristic: depending on the facial orientation, the facial area available for detection becomes narrower, and the likelihood of decreased detection accuracy increases. This characteristic largely depends on the user's mounting position of the camera body 1.

[0772] Therefore, in this embodiment, the detection accuracy (reliability) of the facial direction is calculated based on the mounting position of the camera body 1 and the detection results of the facial direction. When the reliability is high, the facial direction is mostly reflected in the observation direction, and when the reliability is low, other information is mostly reflected in the observation direction. Accordingly, video recording that reflects the user's intent can be performed.

[0773] Figure 38 It is a substitute Figure 35The flowchart illustrates the observation direction determination process performed according to this embodiment when acquiring the facial orientation, as shown in step S6004. This process is performed by the overall control CPU 101 (observation direction determination unit).

[0774] First, in step S7001, the overall control CPU 101 (first observation direction calculation unit, reliability calculation unit) calculates the face direction θ obtained by the face direction detection unit 20 when capturing the nth frame. n The reliability T of the facial orientation is calculated using the first observation direction. n .

[0775] The reliability T in the facial orientation is calculated as follows: n .

[0776] First, adjust the facial orientation θ n Divided into including facial direction θ yaw θ pitch and θ roll The three components. Here, the facial direction θ yaw It is the rotational component that causes horizontal movement of the face, with the facial direction θ. pitch It is the rotational component that causes vertical movement of the face, and the facial direction θ roll It is the rotational component that causes the head to tilt.

[0777] In this embodiment, it is assumed that the camera body 1 is mounted in the user's collarbone and the facial direction is detected from below the face. Therefore, the following formula 701 is used to obtain T. n (0≤Tn≤1).

[0778] [Mathematical Expression 2]

[0779]

[0780] Figure 39 Showing the facial direction θ yaw Reliability of facial orientation T n The value of θ changes, and represents the facial direction. yaw The greater the change in facial orientation relative to the frontal view, the higher the reliability T of the facial orientation. n The more it drops.

[0781] In this embodiment, although Equation 701 is used to calculate the facial orientation reliability T n However, a weighted average of the facial direction reliability calculated by weighting previously calculated facial direction reliability based on factors such as the detection accuracy and detection frame rate of the facial direction using the facial direction detection unit 20 can be used. Additionally, T can be calculated... n The accuracy of pattern matching and equipment location are weighted in time.

[0782] Furthermore, in this embodiment, the determination coefficient obtained from Equation 701 is used to calculate the reliability of the predicted subject area. However, the method for calculating the determination coefficient is not limited to this. For example, a determination coefficient adjusted based on the equipment position that can be estimated through calibration according to Embodiment 2 can be used. Additionally, if the detection accuracy is determined to be low during calibration, the determination coefficient can be changed based on the accuracy. Furthermore, when using machine learning to detect facial orientation, the correlation ratio can be reflected in the determination coefficient.

[0783] In step S7002, the overall control CPU 101 obtains the angular velocity A of the facial direction as follows. More specifically, it uses the following equation 702, based on the facial direction θ obtained from the facial direction detection unit 20 during the capture of the nth frame. n Facial orientation acquisition time t n The facial orientation information θ from a previous frame stored in main memory 103 n-1 and the time t to which it is obtained n-1 To obtain the angular velocity A.

[0784] [Mathematical Expression 3]

[0785]

[0786]

[0787] In this embodiment, although the angular velocity A is calculated using information from the current frame and the frame preceding it, the angular velocity can be obtained using one or more pieces of past information, such as the frame rate.

[0788] In step S7003, the overall control CPU 101 (observation direction prediction unit) predicts the current facial direction based on past facial direction changes stored in the main memory 103. In this embodiment, the current frame is used as a reference, and the four frames past it are set as a predetermined time period. If the facial direction is continuously shifted in the same constant direction three or more times within these four frames, it is determined that the observation direction can be predicted based on the past facial direction and angular velocity. Furthermore, in performing this prediction, the predicted angular velocity B, which is a weighted average of the angular velocities obtained from the past four frames, is calculated using Equation 703, and the predicted facial direction θ is calculated using Equation 704. ave (Second observation direction). The calculations of equations 703 and 704 are respectively related to... Figure 40 The processing shown in (a1) and (a2) corresponds to that.

[0789] The length of the predetermined time period used in step S7003 and the weighted average method can be changed according to the frame rate and the detection accuracy of the face direction detection unit 20.

[0790] [Mathematical Expression 4]

[0791]

[0792]

[0793]

[0794] In step S7004, the overall control CPU 101 uses internal information stored in the main memory 103, excluding the information supplied from the face direction detection unit 20, to predict the observation direction. More specifically, in this embodiment, the current state of the subject being tracked is determined based on the subject detection history. If it is determined that the subject is being tracked, the predicted observation direction θ based on the movement of the subject is calculated. sub (Second observation direction). In this embodiment, the current frame is used as a reference, and the four frames preceding it are set as a predetermined time period. If the same subject is detected three or more times in the video captured from the corresponding four frames, it is determined that the subject is being tracked. In the subject tracking determination, the determination criteria can be changed based on the detection cycle and detection accuracy based on the overall control CPU 101. Subject tracking detection technology is known, therefore its detailed description will be omitted.

[0795] In this embodiment, although the internal information used for predicting the observation direction in step S7004 is the subject detection history, it is not limited to this. For example, based on the mounting position and capabilities of the camera body 1, the prediction of the observation direction can be made using the user's facial information shown in the imaging unit 40 and information related to the movement and posture of the camera body 1 detected by the angular velocity sensor 107 and the accelerometer 108. Furthermore, similar to step S6006 in embodiment 6, when a pre-registered subject exists, the overall control CPU 101 (third observation direction prediction unit) can obtain the direction of the pre-registered subject in the latest captured video as the predicted observation direction θ. sub .

[0796] In step S7005, the overall control CPU 101 stores information related to face orientation detection as history in the main memory 103. Here, the information related to face orientation detection consists of the following: the angular velocity A of the face orientation generated in step S7002, and the face orientation reliability T calculated in step S7001. nThe facial direction θ detected by the facial direction detection unit 20 n Facial orientation acquisition time t n And information indicating the time point in time when this information was generated.

[0797] In step S7006, the facial orientation reliability T calculated in step S7001... n If the value is equal to or higher than the predetermined value, the overall control CPU 101 determines that the reliability of the facial orientation is high and proceeds to step S7009.

[0798] In step S7009, the overall control CPU 101 determines the facial orientation as the current observation orientation θ'. n And proceed to step S7013.

[0799] On the other hand, the facial orientation reliability T calculated in step S7001 n If the value is lower than the predetermined value (No in step S7006), the process proceeds to step S7007.

[0800] In step S7007, the predicted facial direction θ can be estimated in step S7003. ave And satisfy |θ n -θ ave If the conditions are within a predetermined angle, the process proceeds to step S7010. In this embodiment, π / 8 is used as the predetermined angle for determination.

[0801] In step S7010, the overall control CPU 101 (first observation direction prediction unit) uses θ n and θ ave and facial orientation reliability T n To determine the current observation direction θ' n In this embodiment, the current observation direction θ' is obtained using the following equation 705. n And proceed to step S7013. The calculation of Equation 705 and... Figure 40 The process shown in B corresponds to this. For example... Figure 39 As shown, the facial direction θ yaw The smaller the absolute value of the angle, the higher the reliability T of the facial orientation. n The higher the angle, the higher the facial direction θ, as shown in Equation 705. n Most of it is reflected in the current observation direction θ' n In the middle. On the other hand, facial direction θ yaw The larger the absolute value of the angle, the higher the reliability T of the facial orientation. n The lower the value, therefore, as shown in Equation 705, except for the facial direction θ yaw Other information besides (predicting facial orientation θ)ave Most of this is reflected in the current observation direction θ' n middle.

[0802] [Mathematical Expression 5]

[0803] θ′ n =T n ×θ n +(1-T n )×θ ave …(Formula 705)

[0804] If the conditions in step S7007 are not met, the process proceeds to step S7008. This is done when the predicted observation direction θ can be estimated. sub And satisfy |θ n -θ sub If the conditions are within the predetermined angle, the process proceeds to step S7011. Similar to step S7010, in this embodiment, π / 8 is used as the predetermined angle for determination.

[0805] In step S7011, the overall control CPU 101 (second observation direction prediction unit) uses the facial direction θ n Predict the observation direction θ sub Reliability of facial orientation T n To determine the current observation direction θ' n In this embodiment, the current observation direction θ' is obtained using the following equation 706. n And proceed to step S7013. Similar to step S7010, as follows: Figure 39 As shown, the facial direction θ yaw The smaller the absolute value of the angle, the higher the reliability T of the facial orientation. n The higher the angle, the higher the facial direction θ, as shown in Equation 706. n Most of it is reflected in the current observation direction θ' n In the middle. On the other hand, facial direction θ yaw The larger the absolute value of the angle, the higher the reliability T of the facial orientation. n The lower the value, therefore, as shown in Equation 706, except for the facial direction θ yaw Other information besides (predicting the observation direction θ) sub Most of this is reflected in the current observation direction θ' n middle.

[0806] [Mathematical Expression 6]

[0807] θ′ n =T n ×θ n +(1-T n )×θ sub …(Formula 706)

[0808] If the above conditions are not met in step S7008, it is determined that the observation direction cannot be reliably obtained in the current situation, and the process proceeds to step S7012.

[0809] In step S7012, the current observation direction θ' is... n Determined by the previous observation direction θ' n-1 Add inertia based on the shift of the previous observation direction to make the previous observation direction θ' n-1 The observed direction is moved, the viewing angle is set to be wider than the defined value, and the process proceeds to step S7013. This reduces the possibility of missing the subject the user wants to capture.

[0810] In this embodiment, although based on the facial orientation reliability T n The current observation direction θ' is calculated by switching the detection state of the subject. n The method is applicable, but the configuration is not limited to this. For example, when calculating the predicted facial orientation θ ave and predict the observation direction θ sub Each time, its reliability (predicted direction reliability) is also calculated, and the calculated observation direction θ' can be corrected based on the reliability calculated in this way. n .

[0811] Furthermore, if the calculated reliability is equal to or lower than a predetermined value, the possibility of missing the subject the user wants to capture is considered to be increased; therefore, it is preferable to set the viewing angle to a wide angle wider than the defined viewing angle. In this case, the process can proceed to step S7012. Moreover, subsequently, if one of the calculated reliability values ​​becomes greater than the predetermined value, it is preferable to return the viewing angle to the defined viewing angle.

[0812] according to Figure 38 The treatment shown has a reliability T in the facial direction. n In cases of high elevation, adjust the facial orientation θ. n The current observation direction θ' is determined. n On the other hand, in terms of reliability T in the facial direction n In low cases, depending on the situation, by using facial orientation reliability T n Information from high-point times and other information besides facial orientation are used to determine the current observation direction θ'. n (Record direction), and further widen the field of view as needed.

[0813] In other words, in this embodiment, the reliability T in the face direction n In cases where the accuracy of facial orientation detection is low and predicted to be low, the facial orientation θ is predicted using...ave and predict the observation direction θ sub This can prevent unwanted videos from being captured due to facial orientation detection failure.

[0814] Example 8

[0815] In Example 8, reference will be made to Figures 41A to 41C , Figures 42A to 42C , Figures 43A to 43C , Figures 44A to 44C and Figures 45A to 45G This describes a method for mounting the camera body 1 in a stable position.

[0816] Basically, this embodiment will be described as a derivative of Embodiment 1. For this reason, the same reference numerals used for components of the camera system according to Embodiment 8 that are the same as those used for components of the camera system according to Embodiment 1 will be omitted from repeated descriptions, and other components will be described in detail with additional notes as needed.

[0817] First, the angle adjustment of the connecting unit 80 (neck hook unit) will be explained.

[0818] Figures 41A to 41C This is an enlarged view showing the appearance of the camera / detection unit 10 as seen from the side.

[0819] Figure 41A This is a diagram showing the connection unit 80 at position Ax0, which serves as the standard position. Figure 41B This is a diagram showing the connection unit 80 at position Ax1 after rotating about the rotation axis OA by an angle θA1 relative to position Ax0. Figure 41C This is a schematic diagram showing the internal mechanism of the angle maintaining part 81 as seen when the exterior of the angle maintaining part 81 is removed.

[0820] like Figure 41C As shown, an angle adjustment mechanism 8100 (neck angle adjustment unit) is arranged inside the angle maintaining part 81.

[0821] The angle adjustment mechanism 8100 consists of an angle adjustment cam 8101 and a locking member 8102. The angle adjustment cam 8101 adjusts the angle of the angle holding part 81 relative to the camera / detection unit 10, and the locking member 8102 locks the angle adjustment cam 8101. Figure 41A The OA shown is the rotation center of the angle maintaining part 81, and it exists at the center of the angle adjusting cam 8101.

[0822] The locking member 8102 is a member that is biased to the angle adjusting cam 8101 using a spring (not shown in the figure). When the angle adjusting button 85 is pressed... Figure 2FDuring this period, the bias is released, and the locking member 8102 disengages from the angle adjustment cam 8101. In other words, the angle holding part 81 of the connecting unit 80 is rotatable relative to the camera / detection unit 10 only while the angle adjustment button 85 is pressed.

[0823] By rotating the angle holding part 81 relative to the camera / detection unit 10 while the angle adjustment button 85 is pressed, the user can change the position of the connecting unit 80 from position Ax0 ( Figure 41A Adjust to position Ax1 ( Figure 41B ).

[0824] As a mechanism for maintaining the angle of the angle maintaining section 81 relative to the camera / detection unit 10, although a mechanism for graded adjustment using the angle adjusting cam 8101 and the locking member 8102 is used in this embodiment, a mechanism capable of stepless adjustment using sliding resistance can be used.

[0825] Furthermore, although this embodiment employs a structure where the user rotates the angle holding portion 81 while pressing the angle adjustment button 85, the configuration is not limited to this. For example, a configuration that does not require the angle adjustment button 85 can be used (whereby a balance for rotating the angle holding portion 81 is achieved when an external force equal to or greater than a certain threshold is applied to the angle holding portion 81), for example, using a ball instead of the locking member 8102, or using a sliding resistance configuration.

[0826] Figures 42A to 42C This is a side view showing the appearance of the user wearing the camera body 1.

[0827] Figure 42A This diagram shows a user wearing the camera body 1 with the connection unit 80 at position Ax0 and the strap 82 adjusted to its length. Figure 41B This diagram shows a user wearing the camera body 1 with the connection unit 80 at position Ax0 and the strap 82 adjusted to be short. Figure 41C This diagram shows a user wearing the camera body 1 with the connection unit 80 in position Ax1 and the strap 82 adjusted to be short.

[0828] like Figure 42A and Figure 42C As shown, with the relationship between the position of the connecting unit 80 and the length of the strap 82 in a manner suitable for the user, the camera lens 16 faces the user's front. On the other hand, as... Figure 42B As shown, when the relationship between the position of the connecting unit 80 and the length of the strap 82 is unsuitable for the user, the camera lens 16 is not facing the user's front. Figure 42B In this case, the optical axis of the camera lens 16 is oriented upwards.

[0829] Thus, since the connecting unit 80 is configured to be adjustable, the user can wear the camera body 1 so that the optical axis of the camera lens 16 is approximately parallel to the user's natural line of sight. Furthermore, it is clear that appropriate mounting can be performed when the camera body 1 is positioned so that the optical axis of the camera lens 16 is horizontal.

[0830] Next, we will explain the angle adjustment of the chest connecting pad 18.

[0831] Figures 43A to 43C This is an enlarged view showing the appearance of the camera / detection unit 10 as seen from the side when the connection unit 80 is in a non-display state.

[0832] Figure 43A This is a diagram showing the chest connecting pad 18 in position Bx0 as the standard location, and... Figure 43B This is a diagram showing the chest connecting pad 18 at position Bx1 after rotating about the rotation axis B1 by an angle θB1 relative to position Bx0. Figure 43C This is a schematic diagram showing the internal mechanism of the camera / detection unit 10 as seen when the exterior of the camera / detection unit 10 is removed.

[0833] like Figure 43C As shown, an angle adjustment mechanism 8200 (grounding angle adjustment unit) is arranged inside the camera / detection unit 10.

[0834] The angle adjustment mechanism 8200 consists of an angle adjustment cam 8201 and a locking member 8202. The angle adjustment cam 8201 adjusts the angle of the chest connecting pad 18 relative to the camera / detection unit 10, and the locking member 8202 locks the angle adjustment cam 8201. Figure 43C The OB shown is the rotation center of the chest connecting pad 18.

[0835] The locking member 8202 is a member that is biased to the angle adjustment cam 8201 using a spring (not shown in the figure). During the pressing of the angle adjustment button 8203, the bias is released, and the locking member 8202 disengages from the angle adjustment cam 8201. In other words, the chest connecting pad 18 is rotatable relative to the camera / detection unit 10 only during the pressing of the angle adjustment button 8203.

[0836] By rotating the chest connecting pad 18 relative to the camera / detection unit 10 while pressing the angle adjustment button 8203, the user can adjust the position of the chest connecting pad 18 from position Bx0 to position Bx1.

[0837] As a mechanism for maintaining the angle of the chest connecting pad 18 relative to the camera / detection unit 10, although a mechanism for graded adjustment using the angle adjustment cam 8201 and the locking member 8202 is used in this embodiment, a mechanism capable of stepless adjustment using sliding resistance can be used.

[0838] Furthermore, although this embodiment employs a configuration where the user rotates the breast connecting pad 18 while pressing the angle adjustment button 8203, the configuration is not limited to this. For example, a configuration that does not require the angle adjustment button 8203 (whereby a balance for rotating the breast connecting pad 18 is achieved when an external force equal to or greater than a certain threshold is applied to the breast connecting pad 18) can be used, for example, by using a ball instead of the locking member 8202, or by using a sliding resistance configuration.

[0839] Figures 44A to 44C This is a side view showing the appearance of the user wearing the camera body 1 when the connection unit 80 is in a non-display state.

[0840] Figure 44A This is a diagram showing the appearance of the camera body 1 with the chest erect, wearing the chest connecting pad 18 in position Bx0. Figure 44B This is a diagram showing the appearance of the camera body 1 with a flat chest wearing a chest-connecting pad 18 in position Bx0. Figure 44C This is a diagram showing the appearance of the camera body 1 with a flat chest, wearing a chest-connecting pad 18 in position Bx1.

[0841] like Figure 44A and Figure 44C As shown, when the chest connecting pad 18 is positioned at an angle suitable for the user's chest, the chest connecting pad 18 is grounded over a wide range relative to the user's chest. On the other hand, as... Figure 44B As shown, when the position of the chest connecting pad 18 is not in an angled position suitable for the user's chest, the chest connecting pad 18 is grounded within a small area relative to the user's chest. Figure 44B As shown, when the area of ​​the chest connection pad 18 grounding relative to the user's chest is small, the camera / detection unit 10 can easily move relative to the user's body when the user moves his or her body, and the video recording will shake significantly.

[0842] Thus, since the chest connecting pad 18 is configured to be angle-adjustable, the user can wear the camera body 1 so that the chest connecting pad 18 is mounted over a wide range relative to his or her chest, and can suppress blurring of the captured video.

[0843] In this embodiment, although the chest connecting pad 18 is arranged in the camera / detection unit 10, the same effect can be achieved if the chest connecting pad 18 is arranged in the connecting unit 80. In this case, for example, a similar device is arranged within the connecting unit 80. Figure 41C The angle adjustment mechanism 8100 shown is a similar mechanism for adjusting the angle of the chest connecting pad 18 relative to the connecting unit 80.

[0844] Next, the structure of the belt section 82 and the cable 84 will be described in detail.

[0845] As described in Embodiment 1, the battery unit 90 (power supply unit) and the imaging / detection unit 10 of the camera body 1 are separately configured modules that are electrically connected by a cable 84.

[0846] Here, with the cable 84 and the strap 82 configured separately, it is not preferable from the perspective of the appearance of the camera body 1, nor is it preferable from the perspective of the increased complexity of operation when the user wears the camera body 1 around their neck. Therefore, it is preferable that the strap 82 and the cable 84 are configured as an integrated unit, but the configuration is not limited to this. Figure 2B The configuration shown.

[0847] Figures 45A to 45G This is a diagram showing the connecting surface 83, which is the cut surface of the belt section 82 and the cable 84 integrally configured with the belt section 82.

[0848] Figures 45A to 45C This illustrates the case where cable 84 is constructed using a flexible printed circuit board (FPC), and... Figures 45D to 45G This shows the case where cable 84 is constructed using thin wire.

[0849] exist Figure 45A and Figure 45D In this configuration, when viewed from the connecting surface 83, the cable 84 is embedded within the tape portion 82. In this case, the preferred material for the tape portion 82 is an injection-molded elastic component, such as silicone rubber, an elastomer, rubber, or plastic. One method for manufacturing the tape portion 82 and cable 84 as an integrally configured assembly is to insert the cable 84 during injection molding of the tape portion 82. Alternatively, a manufacturing method can be employed where the tape portion 82 is constructed using two components, and the tape portion 82 constructed using two components is formed integrally with the cable 84 disposed between the two components using adhesive or thermal welding. However, as... Figure 45A and Figure 45D As shown, as long as it is possible to manufacture an assembly in which the tape 82 and the cable 84 are integrated, the manufacturing method is not limited to the two manufacturing methods mentioned above.

[0850] exist Figure 45B , Figure 45C and Figure 45EIn the middle, when viewed from the connection surface 83, the cable 84 is in contact with the outer side of the strip 82.

[0851] exist Figure 45B In the strip 82, when viewed from the connecting surface 83, there is no special shape for integration with the cable 84; the cable 84 is bonded to the surface of the strip 82, thus allowing for low-cost manufacturing. However, when the cable 84 (in this case, an FPC) is present on the viewing side, it is not aesthetically desirable for the FPC to be exposed. In this case, to remove the exposure of the FPC, a coating needs to be applied to the outside of the FPC, or a process that can cover the outside with a film is required. Furthermore, if the cable is present on the user's neck contact side, the wearing comfort deteriorates. In this case, to improve the wearing comfort, a coating needs to be applied to the outside, or a process that can cover the outside with a film is required.

[0852] exist Figure 45C and Figure 45E In the tape section 82, when viewed from the connecting surface 83, a concave shape 83a is formed to integrate with the cable 84, and the cable 84 is arranged within the concave shape 83a. In this case, when the concave shape 83a is positioned on the side that contacts the user's neck, a good appearance of the camera body 1 is ensured, and when the concave shape 83a is positioned so that the cable 84 does not directly contact the user's neck, a good wearing comfort for the user can be maintained even without special treatment. In addition, if a proper design is applied when manufacturing the tape section 82, no additional cost is required to form the concave shape 83a, so it is also preferable from a cost perspective.

[0853] In addition, Figure 45F and Figure 45G In the middle, with the above Figure 45A and Figure 45D Similarly, when viewed from the connection surface 83, the cable 84 is embedded in the strip 82. Figure 45F The structure is shown with a cable 84 present, and Figure 45G The diagram illustrates a structure with multiple cables 84. This structure differs from... Figure 45A and Figure 45DThe structure shown includes features in the connecting surface 83 that ensure the cross-sectional area of ​​the strap 82. The cross-sectional area of ​​the strap 82 on the connecting surface 83 affects the torsional stiffness and bending stiffness of the strap 82, and this stiffness affects the stability of the camera / detection unit 10 in its body-mounted position when the user wears the camera body 1. In other words, the stability of the camera / detection unit 10 is further improved when the cross-sectional area of ​​the strap 82 in the connecting surface 83 becomes larger and the strap has stronger torsional and bending stiffness. When the protruding side of the cable 84 faces the user's neck contact side, the contact area between the user and the strap 82 becomes smaller, resulting in a poorer wearing experience; therefore, it is preferable that the protruding side of the cable 84 is located on the outward side. Therefore, in Figure 45F and Figure 45G In the shape shown, the shape of the cable 84 protruding from the strip 82 when viewed from the side in the external view, therefore the appearance of the camera body 1 is similar to that of the cable 84 protruding from the strip 82. Figures 45A to 45G The other shapes shown are not preferred, but rather the shape that is suitable from the viewpoint of ensuring the rigidity of the belt 82.

[0854] As stated above, it is believed Figure 45C and Figure 45E The shape shown is the optimal choice in terms of balance between appearance and wearing comfort. Furthermore, when cost and rigidity are important, other options can be adopted. Figures 45A to 45G Other shapes shown.

[0855] As described above, although preferred embodiments of the invention have been illustrated, the invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the concept of the invention.

[0856] Example 9

[0857] In Example 9, reference will be made Figure 46A and Figure 46B To illustrate a variation of a camera system including camera body 1.

[0858] Basically, this embodiment will be described as a derivative of Embodiment 1. For this reason, the same reference numerals used for components of the camera system according to Embodiment 9 that are the same as those used for components of the camera system according to Embodiment 1 will be omitted from repeated descriptions, and other components will be described in detail with additional notes as needed.

[0859] In Embodiment 1, although a general smartphone is used in the display device 800, there are many smartphone manufacturers, and their arithmetic capabilities vary widely. For example, in Embodiment 1, when video from the recording direction cut out of the ultra-wide-angle image is sent to the display device 800, information required for optical correction processing and image stabilization processing is added to the video. Although the display device 800 performs distortion aberration correction and image stabilization correction based on this information, there are also cases where the arithmetic capabilities of the smartphone used by the user as the display device 800 are too low for performing such corrections.

[0860] Therefore, the camera system according to this embodiment has a display device 9800, the arithmetic operation capability of which is lower than that of the camera body 1' which is a camera device and the display device 800.

[0861] In camera body 1', until the main recording of the video is processed ( Figure 7A When the processing up to steps S100 to S600 is completed, steps S800 and S900 are performed without transmitting the video to the display device 9800. After that, in the camera body 1', processing is performed to transmit the video, which has been processed in steps S800 and S900, to the display device 9800.

[0862] On the other hand, in the display device 9800, the video supplied from the camera body 1' is recorded as is without performing the processing steps S800 and S900.

[0863] The camera system according to this embodiment will be described in more detail below.

[0864] Figure 46A This is a block diagram showing the hardware structure of a display device 9800 connected to a camera body 1', which is a camera device according to this embodiment.

[0865] exist Figure 46A In the diagram, the same reference numerals are assigned to... Figure 6 The components shown are the same as those in the hardware structure of the display device 800 according to Embodiment 1, and repeated descriptions are omitted.

[0866] The display device 9800 includes a display device control unit 9801 instead of the display device control unit 801, and does not include the face sensor 806, which is different from the display device 800.

[0867] The display device control unit 9801 is configured using arithmetic operation capabilities. Figure 6The CPU is composed of a low-performance CPU. Furthermore, the capabilities of the built-in non-volatile memory 812 and main memory 813 may be lower than those according to Embodiment 1.

[0868] Figure 46B This is a functional block diagram of the camera body 1'.

[0869] exist Figure 46B In this context, the same reference numerals are assigned to... Figure 4 The functional blocks shown are the same as those in the camera body 1 according to Embodiment 1, and repeated descriptions will be omitted.

[0870] exist Figure 46B Among the functional blocks shown, there is an optical image stabilization unit 9080 that performs optical correction and image stabilization processing. Furthermore, these functional blocks are executed not under the control of the overall control CPU 101, but under the control of the overall control CPU 9101, which differs from... Figure 4 Furthermore, the communication counterpart of the predetermined sending unit 70 is not the display device 800, but the display device 9800, which is also different.

[0871] In other words, in this embodiment, the optical image stabilization unit 9080 of the overall control CPU 9101 uses optical correction values ​​and gyroscope data to perform optical distortion correction and image stabilization correction. For this reason, the amount of data in the video file after optical distortion correction and image stabilization correction sent by the sending unit 70 to the display device 9800 in this embodiment is smaller than the amount of data in the video file 1000 sent by the sending unit 70 to the display device 800 in Embodiment 1.

[0872] Furthermore, in the display device 9800, by omitting steps S800 and S900, the arithmetic operation capability can be suppressed to a level lower than that of the display device 800. Alternatively, the video captured by the camera body 1' can be read using a simple display device 900 (viewing unit) such as a smartwatch.

[0873] As described above, a method for using camera body 1 to cut out and record video of an area observed by a user has been explained. Hereinafter, the area cut out by camera body 1 will be referred to as the cut-out area. In cases where the user causes their face to deviate from its orientation, the cut-out area may deviate from the area the user wishes to record in a portion of the moving image. Embodiments 10 and 11 are embodiments of a method for sending a moving image captured by camera body 1 to display device 800 and correcting the cut-out area.

[0874] Example 10

[0875] Figures 50A to 50EThis is a diagram showing a portion of a moving image. The display device 800 receives a moving image captured by the user in the moving image mode of the camera body 1 and stores the received moving image in a large-capacity non-volatile memory 814. Figures 50A to 50E An example is shown below: For five frames in a moving image, a cutout area 5000 defined by the camera body 1 is displayed on the display device 800.

[0876] Figure 50A The appearance of the subject 131, which the user wishes to keep as a video, is shown in the center of the cut-out area 5000. Figure 50B This shows the appearance of the subject 131 moving to the corner of the cut-out area 5000. Figure 50C This shows the appearance of the subject 131 cut out from the cut-out area 5000. Figure 50D This shows the appearance of subject 131 returning to the center of the cut-out area 5000. Figure 50E This shows the appearance of the subject 131 being able to return to the center of the cut-out area 5000.

[0877] When a child is being photographed as subject 131, it can be assumed that the user's facial orientation deviates from that of subject 131 midway through the photograph, depending on whether the user directs their face toward a nearby passing vehicle or the direction of a sound. In this case, such as Figure 50C As shown, there is a possibility that the subject 131 may be cut out from the cut-out area.

[0878] In this embodiment, in order for the display device 800 to correct the cut-out area of ​​the motion image captured by the camera body 1 after shooting, the camera body 1 has a full-area motion image mode as a shooting mode for recording the captured motion image without performing cut-out processing.

[0879] refer to Figure 51 This will explain the shooting mode of camera body 1. Figure 51 This is a diagram of the camera / detection unit 10 according to Embodiment 10, as seen from the back side (the side in contact with the user). The structure, except for the camera mode switch 12A, is similar to... Figure 2C The structure shown is similar, so its description will be omitted. The camera mode switch 12A is a slider-type switch, which, in addition to being used to select... Figure 2C In addition to the same "Photo", "Normal" and "Pri" camera modes shown, you can also select "Overall" which corresponds to the motion picture mode for the entire area.

[0880] refer to Figures 52A to 52DThe process described herein is for recording a motion image of the entire area and sending the motion image to a display device 800 without cutting out a region from the motion image captured by the camera body 1. Figures 52A to 52D The processing shown is the same as Figure 7A , Figure 7B , Figure 7E and Figure 14 The same reference numerals are assigned to the same processes as shown in Embodiment 1, and their detailed descriptions are omitted.

[0881] In embodiment 10, the camera mode switch 12A is set to "overall," and the camera body 1 captures images in the entire area motion image mode. The camera body 1 records the motion image of the entire area without cutting out and sends the recorded motion image to the display device 800.

[0882] exist Figure 52A In step S8100 shown, the overall control CPU 101 performs... Figure 7A The preparation operation process corresponding to step S100 shown. Figure 52B This is a flowchart of the subroutine for the preparation operation in step S8100. Figure 52B In the preparation operation process shown, in step S8102, if the mode selected by the camera mode switch 12A is the whole area motion image mode, the process proceeds to step S103, similar to the case of selecting the motion image mode.

[0883] In step S103, the overall control CPU 101 reads various settings for the entire area motion picture mode from the built-in non-volatile memory 102 and stores the read settings in the main memory 103. These settings include settings such as image stabilization level. In step S104, the overall control CPU 101 starts the operation of the camera driver 41 for the entire area motion picture mode and ends the preparation operation processing.

[0884] exist Figure 52A In steps S200 to S400 shown, the overall control CPU 101 performs... Figure 7A The processes in steps S200 to S400 are similar. In step S400, the camera unit 40 captures an image and generates captured image data.

[0885] In step S8500, the image cropping / display processing unit 50 performs display processing on the entire area of ​​the captured image data generated in step S400. Figure 52C This is a flowchart of the subroutine for the whole area imaging processing in step S8500. Figure 52CThe entire area imaging process shown is performed by... Figure 7E The processing obtained by the exclusion region cutting-out process (step S502) in the process according to Embodiment 1 is shown.

[0886] In step S501, with Figure 7E Similar to step S501, the image cropping / development processing unit 50 acquires the raw data of the entire region. In steps S8503 to S8508, the image cropping / development processing unit 50 processes the entire region... Figure 7E The processing of steps S503 to S508 shown is similar to that of the previous steps.

[0887] exist Figure 52A In step S8600, the overall control CPU 101 stores the video of the entire area after imaging in step S8500 in the main recording unit 60. Figure 52D This is a flowchart of the main record processing subroutine in step S8600. Figure 52D The main record processing shown will be explained in conjunction with... Figure 14 The main record processing shown in Example 1 is different for different points. Figure 52D The master record processing shown is related to Figure 14 The processes corresponding to steps S601 to S606 shown are performed for each frame of the video after it has been displayed in step S8500. Figure 52D In this context, the frame that is the object of processing will be referred to as the current frame.

[0888] exist Figure 52D In step S601 shown, with Figure 14 Similar to step S601 shown, the overall control CPU 101 appends (adds) information about the cutout position (position of the cutout area) of the current frame's image to the correction data, which serves as metadata for the current frame. For example, the added cutout position information is... Figure 7D The coordinates Xi and Yi of the video recording frame 127i recorded in step S305 are shown.

[0889] In step S8601, the overall control CPU 101 also appends (adds) the cropping size (size of the cropped area) of the current frame's image to the correction data. For example, the appended cropping position information is... Figure 7D The horizontal width WXi and vertical width WYi of the video recording frame 127i recorded in step S305 are shown.

[0890] Processing in steps S602 to S605 Figure 14The processing shown is similar, with the same reference numerals assigned to it. In step S8606, the correction data (frame metadata) for the current frame includes not only the position of the cut-out region but also its size. The overall control CPU 101 processes each frame of the video after display in step S8500. Figure 52D The processing shown generates a video file for transmission to the display device 800. By including the position and size of the cut-out regions of each frame in the video file, the display device 800 can perform cut-out processing on the video file received from the camera body 1.

[0891] refer to Figure 53 This will explain the data structure (format) of the video file. Figure 15 The video file shown is similar to 1000. Figure 53 The video file 2000 (moving picture data) shown includes a header 1001 and frame 2002. Frame 2002 consists of a frame dataset corresponding to the total number of frames in the moving picture, in which the images of each frame constituting the moving picture and the corresponding frame metadata are formed as a set. The images of each frame in the video file 2000 are images of the entire region.

[0892] The frame metadata for each frame includes the cutout size information appended as correction data in step S8601. Additionally, the frame metadata for video file 2000 includes reference data... Figure 15 In addition to the aforementioned parameters, three other parameters are included: cutout position correction amount, cutout size correction amount, and user-defined operation flag. These three added parameters are used by the display device 800 to process the cutout area of ​​the moving image.

[0893] The user-modified operation flag indicates whether the cutout area has been corrected based on the user's operation. The cutout position correction amount is the amount by which the position of the cutout area is corrected based on the user's operation. The cutout size correction amount is the amount by which the size of the cutout area is corrected based on the user's operation.

[0894] In the main recording process of step S8600, the cutout position correction amount and cutout size ...

Claims

1. A display device, comprising: An acquisition unit is configured to acquire motion image data, which is acquired by adding information of a cutout region of the motion image set based on the facial orientation of the user capturing the motion image to the motion image. A display control unit is configured to display a box representing the cut-out area when reproducing the motion image data; A receiving unit configured to accept operations for correcting the cut-out area of ​​the frame; A correction unit is configured to add correction information representing the amount of correction of the cut-out region after correction by the operation to the motion image data; as well as A generation unit is configured to generate a cut-out motion image from the motion image based on information about the cut-out region and the correction information. The correction information includes at least one of a cutout position correction amount and a cutout size correction amount. The cutout position correction amount is a correction amount for the position of the cutout area after user operation correction, and the cutout size correction amount is a correction amount for the size of the cutout area after user operation correction. The generation unit generates the cut-out motion image by correcting the cut-out region of the unoperated frame, which is a frame that has not received the operation, using correction information for the correction frame, which is a frame that has received the operation.

2. The display device according to claim 1, wherein, The generation unit corrects the cut-out region of the non-operation frame by interpolating between correction information of a first correction frame placed before the non-operation frame in the correction frame and correction information of a second correction frame placed after the non-operation frame in the correction frame.

3. The display device according to claim 2, wherein, The first correction frame is the initial frame of the motion picture, or the initial frame of a scene that includes the unoperated frame among multiple scenes included in the motion picture.

4. The display device according to claim 2 or 3, wherein, The second correction frame is the last frame of the motion picture, or the last frame of a scene that includes the unoperated frame among the multiple scenes included in the motion picture.

5. The display device according to claim 1, wherein, The generation unit corrects the cut-out region of the non-operated frame by interpolating the correction information of the frames before a predetermined first frame number for the non-operated frame and the correction information of the frames after a predetermined second frame number for the non-operated frame.

6. The display device according to claim 2 or 3, wherein, The generation unit uses at least one of the following methods to correct the cut-out region of the unoperated frame: linear interpolation, cubic interpolation based on fade-in and fade-out functions, and multi-point interpolation.

7. The display device according to any one of claims 1 to 3, wherein, The correction of the cut-out region includes a change in at least one of the position and size of the cut-out region.

8. The display device according to any one of claims 1 to 3, wherein, The correction information includes a user-changed operation flag, which indicates whether the cut-out area has been corrected through user operation.

9. The display device according to any one of claims 1 to 3, wherein, The operation is to drag the box to the moving destination of the cut-out area.

10. The display device according to any one of claims 1 to 3, wherein, The operation is a tap on the cut-out area to move to the destination or to the subject being photographed.

11. The display device according to any one of claims 1 to 3, wherein, The operation is a pinching or pinching action on the frame.

12. The display device according to any one of claims 1 to 3, wherein, The operation is the act of touching the moving destination of the frame or the subject being photographed during the reproduction of the moving image.

13. The display device according to any one of claims 1 to 3, wherein, The operation is the pinching or splitting of the frame during the reproduction of the moving image.

14. The display device according to claim 12, wherein, The motion image is reproduced at a speed lower than normal.

15. The display device according to any one of claims 1 to 3, wherein, The motion image data includes audio data collected by multiple microphones during the capture of the motion image, and The generation unit adjusts the balance of the voice data collected by the multiple microphones based on the correction of the cut-out region.

16. A method for controlling a display device, the method comprising: Acquire motion image data by adding information about the cutout region of the motion image, which is set based on the facial orientation of the user who took the motion image, to the motion image. When reproducing the motion image data, a box representing the cut-out region is displayed; Accept operations for correcting the cut-out area of ​​the frame; Add correction information, representing the correction amount of the cut-out region after correction by the operation, to the motion image data; as well as A cut-out motion image is generated from the motion image based on the information of the cut-out region and the correction information. The correction information includes at least one of a cutout position correction amount and a cutout size correction amount. The cutout position correction amount is a correction amount for the position of the cutout area after user operation correction, and the cutout size correction amount is a correction amount for the size of the cutout area after user operation correction. In the generation process, the cut-out motion image is generated by correcting the cut-out region of the unoperated frame, which is a frame that has not received the operation, using correction information for the correction frame, which is a frame that has received the operation.

17. A computer-readable storage medium for storing a program that causes a computer to function as a unit of a display device according to any one of claims 1 to 15.

18. A computer program product comprising a program for causing a computer to function as a unit of a display device according to any one of claims 1 to 15.

Images