Imaging apparatus and control method for the same

Abstract

To provide an imaging apparatus that can use appropriate calibration data even when its user changes, and a control method for the same.SOLUTION: An imaging apparatus has a function of detecting the direction of a user's line of sight. The imaging apparatus has storage means that stores calibration data for correcting individual difference between users for a result of detection of the line-of-sight direction. The imaging apparatus receives, from an external device located within a communication rage, identification information of the external device, and acquires the calibration data associated with the identification information from the storage means. The imaging apparatus corrects the result of detection of the line-of-sight direction by using the acquired calibration data.SELECTED DRAWING: Figure 24A

Description

[Technical Field]

[0001] The present invention relates to an imaging device and a control method thereof, and more particularly to an imaging device that uses individual calibration data and a control method thereof. [Background technology]

[0002] When shooting handheld, the photographer must constantly pay attention to the camera's shooting direction and angle of view in order to capture the desired scene. This makes it difficult for the photographer to truly enjoy the scene they are capturing. In addition, since at least one hand must be kept on the camera, their movements while shooting are restricted.

[0003] To address this issue, an imaging device has been proposed that is worn around the photographer's neck and captures images in the same direction as the photographer's line of sight (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-140328 Summary of the Invention [Problem to be solved by the invention]

[0005] The imaging device proposed in Patent Document 1 does not need to be held by the photographer in his / her hands. In addition, since the imaging device captures images in the photographer's observation direction, it is possible to prevent the photographer from being distracted by taking pictures and being unable to enjoy the scene.

[0006] However, in order to achieve photography that matches the photographer's intentions with the imaging device proposed in Patent Document 1, it is necessary to perform calibration to generate correction data (calibration data) that corresponds to individual differences such as the photographer's body shape, etc. Furthermore, performing calibration every time a different user changes is cumbersome.

[0007] The present invention has been made in view of the above problems in the prior art, and in one embodiment, the present invention provides an imaging apparatus and a control method thereof that allow appropriate calibration data to be used even if the user changes. [Means for solving the problem]

[0008] In one aspect, the present invention provides an imaging device having a function of detecting a user's gaze direction, the imaging device comprising: storage means for storing calibration data for correcting individual differences among users in the gaze direction detection result; receiving means for receiving identification information of an external device from an external device present within a communication range; acquisition means for acquiring calibration data associated with the identification information from the storage means; and correction means for correcting the gaze direction detection result using the acquired calibration data. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an imaging apparatus and a control method thereof that allow appropriate calibration data to be used even if the user changes. [Brief explanation of the drawings]

[0010] [Figure 1A] 1 is an external view of a camera body including an imaging and detection unit as an imaging device according to a first embodiment. [Figure 1B] FIG. 10 is a diagram showing the camera body hung by a user. [Figure 1C] 1B is a view of the battery section in the camera body as seen from the rear of FIG. 1A. FIG. [Figure 1D] 1 is an external view of a display device as a mobile device according to a first embodiment, which is configured as a separate body from a camera body. [Figure 2A] FIG. 2 is a front view of the imaging and detection section of the camera body. [Figure 2B] 10A and 10B are diagrams showing the shape of a band portion of a connecting portion in a camera body. [Figure 2C]This is a view of the imaging and detection unit from the back. [Figure 2D] This is a top view of the imaging and detection unit. [Figure 2E] FIG. 2 is a diagram showing the configuration of a face direction detection section disposed inside the imaging / detection section, below a face direction detection window in the camera body. [Figure 2F] FIG. 2 is a view of the camera body worn by a user, as seen from the left side of the user. [Figure 3] FIG. 2 is a diagram illustrating the battery section in detail. [Figure 4] FIG. 2 is a functional block diagram of the camera body according to the first embodiment. [Figure 5] FIG. 2 is a block diagram showing the hardware configuration of the camera body. [Figure 6] FIG. 2 is a block diagram showing a hardware configuration of the display device. [Figure 7A] 5 is a flowchart showing an outline of an image capturing and recording process according to the first embodiment, which is executed in the camera body and the display device. [Figure 7B] 7B is a flowchart of a subroutine of the preparatory operation process of S100 in FIG. 7A according to the first embodiment. [Figure 7C] 7B is a flowchart of a subroutine of the face direction detection process of S200 in FIG. 7A according to the first embodiment. [Figure 7D] 7B is a flowchart of a subroutine of the printing direction / range determination process of S300 in FIG. 7A according to the first embodiment. [Figure 7E] 7B is a flowchart of a subroutine of the recording area development process of S500 in FIG. 7A according to the first embodiment. [Figure 7F] FIG. 7B is a diagram for explaining the processing from S200 to S600 in FIG. 7A in the moving image mode. [Figure 8A] FIG. 10 is a diagram showing an image of the user seen through a face direction detection window. [Figure 8B] 10A and 10B are diagrams illustrating a case where a fluorescent light in a room is reflected as a background in an image of a user seen through a face direction detection window. [Figure 8C]8B and a fluorescent light in the background are imaged on the sensor of the infrared detection processing device through a face direction detection window without turning on the infrared LED of the infrared detection processing device. [Figure 8D] 8C is a diagram showing an image of the user shown in FIG. 8B and a fluorescent lamp in the background, which is formed on the sensor of the infrared detection processing device through a face direction detection window with the infrared LED turned on. FIG. [Figure 8E] 8C is a diagram showing a differential image calculated by subtracting the image of FIG. 8C from the image of FIG. 8D. [Figure 8F] This is a diagram showing the result of adjusting the shading of the difference image in Figure 8E to match the scale with the light intensity of the reflected infrared light beam projected onto the face and neck of the user. [Figure 8G] FIG. 8F is a diagram in which symbols indicating each part of the user's body, as well as symbols of a double circle indicating the neck position and a black circle indicating the chin position, are superimposed. [Figure 8H] 8B is a diagram showing a difference image calculated in the same manner as in FIG. 8E when the user's face is facing rightward. FIG. [Figure 8I] The shading of the difference image in Figure 8H has been adjusted to match the scale of the reflected infrared light beam projected onto the user's face and neck, and the double circle indicating the neck position and the black circle indicating the chin position have been superimposed. [Figure 8J] FIG. 10 is a diagram showing an image of the user seen through the face direction detection window when the user is facing the face 33° above the horizontal. [Figure 8K] This figure shows the difference image calculated in the same way as in Figure 8E when the user is tilting their face 33° above horizontal, scaled to match the light intensity of the reflected infrared light projected onto the user's face and neck, and superimposed with a double circle indicating the neck position and a black circle indicating the chin position. [Figure 9] 10 is a timing chart showing the timing of lighting an infrared LED and related signals. [Figure 10] 10A and 10B are diagrams illustrating the vertical movement of the user's face. [Figure 11A]10 is a diagram showing the target field of view in an ultra-wide-angle image captured by the imaging unit of the camera body when the user is facing forward. FIG. [Figure 11B] 11B is a diagram showing an image of a target field of view cut out from the super-wide-angle image of FIG. 11A. FIG. [Figure 11C] 10 is a diagram showing a target field of view in an ultra-wide-angle image when a user is observing a subject A. FIG. [Figure 11D] 11D is a diagram showing an image in which distortion and shaking have been corrected for the image of the target field of view in FIG. 11C cut out from the super-wide-angle image. [Figure 11E] 11B is a diagram showing a target field of view in an ultra-wide-angle image when a user is observing subject A at a smaller angle of view setting value than that in FIG. 11C. [Figure 11F] 11E is a diagram showing an image in which distortion and shaking have been corrected for the image of the target field of view in FIG. 11E cut out from the super-wide-angle image. [Figure 12A] FIG. 10 is a diagram showing an example of a target field of view in an ultra-wide-angle image. [Figure 12B] 12B is a diagram showing an example of a target field of view in a super-wide-angle image, which has the same angle of view setting value as the target field of view in FIG. 12A but is observed from a different direction. [Figure 12C] 12B is a diagram showing another example of a target field of view in a super-wide-angle image, which has the same angle of view setting value as the target field of view in FIG. 12A but is observed in a different direction. [Figure 12D] 12D is a diagram showing an example of a target field of view in an ultra-wide-angle image, the target field of view being observed in the same direction as the target field of view in FIG. 12C but with a smaller angle of view setting value. [Figure 12E] 12B is a diagram showing an example in which a preliminary image stabilization area corresponding to a predetermined image stabilization level is provided around the target field of view shown in FIG. 12A. FIG. [Figure 12F] 12E is a diagram showing an example in which a spare image stabilization area corresponding to the same level of image stabilization as the spare image stabilization area in FIG. 12E is added around the target field of view shown in FIG. 12B. [Figure 12G] 12E is a diagram showing an example in which a spare image stabilization area corresponding to the same level of image stabilization as the spare image stabilization area in FIG. 12E is provided around the target field of view shown in FIG. 12D. [Figure 13]FIG. 10 is a diagram showing a menu screen for various settings in video mode, which is displayed on the display unit of the display device before image capture with the camera body. [Figure 14] 7B is a flowchart of a subroutine of the primary recording process of S600 in FIG. 7A. [Figure 15] FIG. 2 is a diagram showing the data structure of a video file generated by the primary recording process. [Figure 16] 7B is a flowchart of a subroutine of the transfer process to the display device in S700 of FIG. 7A. [Figure 17] 7B is a flowchart of a subroutine of the optical correction process in S800 of FIG. 7A. [Figure 18] 18A to 18C are diagrams for explaining the process of performing distortion correction in S803 of FIG. 17. [Figure 19] 7B is a flowchart of a subroutine of the image stabilization process of S900 in FIG. 7A. [Figure 20] FIG. 10 is a diagram showing details of a calibrator used in a calibration process according to the second embodiment. [Figure 21] 10 is a flowchart of a calibration process according to the second embodiment, which is executed in the camera body and the calibrator. [Figure 22A] FIG. 22 is a diagram showing a screen displayed on the display unit of the calibrator in S3103 of FIG. 21 during calibration in the front direction of the user. [Figure 22B] 22B is a diagram showing a state in which a user holds the calibrator in front of them in accordance with the instructions shown on the instruction display in FIG. 22A. FIG. [Figure 22C] FIG. 22C is a schematic diagram showing the entire super-wide-angle image captured by the imaging lens in the state of FIG. 22B. [Figure 22D] FIG. 22D is a schematic diagram showing an image obtained by correcting the aberration of the super-wide-angle image shown in FIG. 22C. [Figure 22E] FIG. 22 is a schematic diagram showing a face direction image acquired by the face direction detection unit in S3108 of FIG. 21 during calibration for the front direction of the user. [Figure 22F]FIG. 22 is a schematic diagram showing the in-camera image displayed in S3107 of FIG. 21. [Figure 23A] FIG. 22 is a diagram showing a screen displayed on the display unit of the calibrator in S3103 of FIG. 21 during calibration of the user's right hand in the upward direction. [Figure 23B] 23B is a diagram showing a state in which a user holds the calibrator to the upper right in accordance with the instructions shown on the instruction display in FIG. 23A. FIG. [Figure 23C] FIG. 23C is a schematic diagram showing the entire super-wide-angle image captured by the imaging lens in the state of FIG. 23B. [Figure 23D] FIG. 23D is a schematic diagram showing an image obtained by correcting the aberration of the super-wide-angle image shown in FIG. 23C. [Figure 23E] FIG. 22 is a schematic diagram showing a face direction image acquired by the face direction detection unit in S3108 of FIG. 21 during calibration of the user's right hand in the upward direction. [Figure 24A] 4 is a flowchart relating to a process of specifying calibration data in the first embodiment. [Figure 24B] 4 is a flowchart relating to a process of specifying calibration data in the first embodiment. [Figure 24C] 4 is a flowchart relating to a process of specifying calibration data in the first embodiment. [Figure 25] 10 is a flowchart relating to a process of specifying calibration data in the second embodiment. [Figure 26] 10 is a flowchart illustrating a process for specifying calibration data in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. ●(First embodiment) 1A to 1D are diagrams illustrating a camera system comprising a camera body 1 including an imaging / detection unit 10 as a wearable imaging device according to this embodiment, and a display device 800 configured separately from the camera body 1. Note that, although this embodiment shows an example in which the camera body 1 and the display device 800 are separate entities, they may also be configured as an integrated unit. Furthermore, a user wearing the camera body 1 will be referred to as a "user" hereinafter.

[0013] FIG. 1A is an external view of the camera body 1. FIG.

[0014] 1A, the camera body 1 includes an imaging / detection unit 10, a battery unit 90, a right-side connector 80R, and a left-side connector 80L. The right-side connector 80R connects the imaging / detection unit 10 and the battery unit 90 on the right side of the user's body (the left side as you face FIG. 1A). The left-side connector 80L connects the imaging / detection unit 10 and the battery unit 90 on the left side of the user's body (the right side as you face FIG. 1A).

[0015] The photographing / detection unit 10 includes a face direction detection window 13, a start switch 14, a stop switch 15, an imaging lens 16, an LED 17, and microphones 19L and 19R.

[0016] The face direction detection window 13 transmits infrared light projected from an infrared LED 22 (FIG. 5) built into the photographing and detection unit 10 for detecting the position of each part of the user's face and its reflected light.

[0017] The start switch 14 is a switch for starting imaging.

[0018] The stop switch 15 is a switch for stopping imaging.

[0019] The imaging lens 16 guides light rays to be captured into the imaging / detection unit 10, and forms an optical image on the solid-state imaging element 42 (FIG. 5).

[0020] The LED 17 is an LED that indicates that imaging is in progress or that a warning is issued.

[0021] Microphones 19R and 19L are microphones that pick up surrounding sounds, with microphone 19L picking up sounds from the left side of the user's surroundings (the right side as viewed in FIG. 1A), and microphone 19R picking up sounds from the right side of the user's surroundings (the left side as viewed in FIG. 1A).

[0022] 1B is a diagram showing a typical state in which camera body 1 is in use. Here, the user wears camera body 1 around the front of their neck by hanging the attachment part consisting of right connector 80R, battery part 90, and left connector 80L around their neck.

[0023] The camera body 1 is attached so that the battery unit 90 is positioned on the back of the user's neck and the imaging / detection unit 10 is positioned on the chest side of the user's neck. This biases and supports the imaging / detection unit 10 toward the chest by the left and right connectors 80L and 80R. This positions the imaging / detection unit 10 between the user's left and right collarbones (near the top of the sternum). At this time, the face direction detection window 13 is positioned under the user's chin. The infrared condenser lens 26, which will be illustrated later in FIG. 2E, is located within the face direction detection window 13. The optical axis (detection optical axis) of the infrared condenser lens 26 is different from the optical axis (imaging optical axis) of the imaging lens 16 and faces upward (toward the user's head). The face direction detection unit 20 (face direction detection means, see FIG. 5) including the infrared condenser lens 26 detects the user's observation direction from the positions of various parts of the face. This enables the imaging unit 40, described later, to capture an image in the user's observation direction.

[0024] How to adjust the setting position depending on individual differences in body shape and clothing will be described later.

[0025] In addition, by placing the imaging / detection unit 10 on the front of the body and the battery unit 90 on the back, the weight is distributed, reducing fatigue for the user and preventing displacement due to centrifugal force when the user moves.

[0026] In this embodiment, the image capturing / detecting unit 10 is attached to the area between the collarbones (the upper end of the sternum) of the user, but this is not limiting. In other words, as long as the face direction detecting unit 20 can detect the observation direction of the user and the image capturing unit 40 can capture an image in the observation direction, the camera body 1 can be attached to any position on the surface of the user's body other than the head.

[0027] FIG. 1C is a view of the battery section 90 as seen from the rear of FIG. 1A.

[0028] 1C, the battery section 90 includes a charging cable insertion port 91, adjustment buttons 92L and 92R, and a notch 93 for preventing the user from touching the spine.

[0029] The charging cable insertion port 91 is an insertion port for a charging cable (not shown), and the charging cable is used to charge internal batteries 94L, 94R (see FIG. 3A) from an external power source and to supply power to the imaging / detection unit 10.

[0030] The adjustment buttons 92L, 92R are buttons for adjusting the lengths of the band portions 82L, 82R of the left and right connection portions 80L, 80R. The adjustment button 92L is a button for adjusting the left band portion 82L, and the adjustment button 92R is a button for adjusting the right band portion 82R. In this embodiment, the adjustment buttons 92L, 92R adjust the lengths of the band portions 82L, 82R independently, but a single button may be used to simultaneously adjust the lengths of the band portions 82L, 82R. Hereinafter, the band portions 82L, 82R are collectively referred to as the band portion 82.

[0031] The spine-preventing cutout 93 is a cutout that avoids the user's spine so that the battery part 90 does not come into contact with the user's spine. By avoiding the protrusion of the human spine, discomfort when wearing the device is reduced, and the device is prevented from moving left and right during use.

[0032] FIG. 1D is an external view of a display device 800 as a mobile device according to the first embodiment, which is configured as a separate body from the camera body 1. FIG.

[0033] 1D, display device 800 includes an A button 802, a display unit 803, a B button 804, an in-camera 805, a face sensor 806, an angular velocity sensor 807, and an acceleration sensor 808. Although not shown in FIG. 1D, display device 800 also includes a wireless LAN capable of high-speed connection with camera body 1.

[0034] The A button 802 is a button that functions as a power button for the display device 800. The display device 800 accepts power ON / OFF operations by pressing and holding the A button 802, and accepts instructions for other processing timing by pressing the A button 802 for a short time.

[0035] The display unit 803 can be used to check images captured by the camera body 1 and display menu screens required for settings. In this embodiment, a transparent touch sensor is also provided on the top surface of the display unit 803, and accepts operations by touching the screen being displayed (for example, the menu screen).

[0036] The B button 804 functions as a calibration button 854 used in the calibration process described later.

[0037] The in-camera 805 is a camera capable of capturing an image of a person viewing the display device 800 .

[0038] The face sensor 806 detects the face shape and viewing direction of a person viewing the display device 800. The specific structure of the face sensor 806 is not particularly limited, but it can be implemented by various sensors such as a structured light sensor, a ToF sensor, or a millimeter wave radar.

[0039] The angular velocity sensor 807 is shown by a dotted line as a perspective view because it is inside the display device 800. The display device 800 of this embodiment also has a calibrator function, which will be described later, and is therefore equipped with gyro sensors in three directions, the X, Y, and Z directions.

[0040] The acceleration sensor 808 detects the attitude of the display device 800 .

[0041] Note that a general smartphone is used as the display device 800 according to this embodiment, and the camera system according to the present invention can be implemented by making the firmware in the smartphone compatible with the firmware on the camera main body 1. However, it is also possible to implement the camera system according to the present invention by making the firmware on the camera main body 1 compatible with the applications and OS of the smartphone that serves as the display device 800.

[0042] 2A to 2F are diagrams for explaining in detail the imaging and detection unit 10. In the following figures, parts that have already been explained are given the same numbers to indicate the same functions, and explanations thereof will be omitted in this specification.

[0043] FIG. 2A is a front view of the imaging and detection unit 10. FIG.

[0044] The right connecting part 80R has an angle holding part 81R made of a hard material that holds the angle with the imaging / detecting part 10, and a band part 82R, and the left connecting part 80L has an angle holding part 81L and a band part 82L.

[0045] 2B is a diagram showing the shapes of the band portions 82L, 82R of the left and right connecting portions 80L, 80R. In this drawing, the angle holding portions 81L, 81R are shown in perspective in order to show the shapes of the band portions 82L, 82R.

[0046] Band part 82L includes a left connection surface 83L that is positioned on the left side of the user's body (the right side as viewed in FIG. 2B) when camera body 1 is worn, and an electrical cable 84. Band part 82R includes a right connection surface 83R that is positioned on the right side of the user's body (the left side as viewed in FIG. 2B) when camera body 1 is worn.

[0047] The left connection surface 83L is connected to the angle retaining portion 81L and has a cross-sectional shape that is not a perfect circle, but is elliptical in this case. The right connection surface 83R also has a similar elliptical shape. The right connection surface 83R and the left connection surface 83L are shaped like the Japanese katakana character "ハ". That is, the distance between the symmetrical portions of the right connection surface 83R and the left connection surface 83L decreases as one moves from bottom to top in FIG. 2B. This ensures that when the user wears the camera body 1 around their neck, the longitudinal axes of the left and right connection surfaces 83L, 83R are aligned with the user's body. This provides a comfortable fit when the band portions 82L, 82R contact the user's body and prevents the imaging / detection unit 10 from moving left and right, forward and backward.

[0048] The electric cable 84 is wired inside the band part 82L and electrically connects the battery part 90 and the imaging / detection part 10. The electric cable 84 connects the power source of the battery part 90 to the imaging / detection part 10 and transmits and receives electric signals to and from the outside.

[0049] 2C is a view from the back of the imaging / detection unit 10. Because FIG. 2C is a view from the side that comes into contact with the user's body, i.e., the opposite side to FIG. 2A, the positional relationship between the right-side connecting part 80R and the left-side connecting part 80L is reversed from that in FIG. 2A.

[0050] The imaging / detection unit 10 has a power switch 11, an imaging mode switch 12, and chest connection pads 18a and 18b on the back side thereof.

[0051] The power switch 11 is a power switch that switches the power supply to the camera body 1 on and off. In this embodiment, the power switch 11 is a switch in the form of a slide lever, but is not limited to this. For example, the power switch 11 may be a push-type switch, or may be a switch that is integral with a slide cover (not shown) of the imaging lens 16.

[0052] The imaging mode switch 12 is a switch for changing the imaging mode and can change modes related to imaging. In this embodiment, the imaging mode switch 12 can switch between still image mode, video mode, and a pre-setting mode set using the display device 800, which will be described later. In this embodiment, the imaging mode switch 12 is a switch in the form of a slide lever that can select one of "Photo," "Normal," and "Pre" shown in FIG. 2C by sliding the lever. The imaging mode switches to the still image mode by sliding to "Photo," to the video mode by sliding to "Normal," and to the pre-setting mode by sliding to "Pre." Note that the imaging mode switch 12 is not limited to the form of this embodiment as long as it is a switch that can change the imaging mode. For example, the imaging mode switch 12 may be configured with three buttons: "Photo," "Normal," and "Pre."

[0053] Chest connection pads 18a and 18b are the portions that come into contact with the user's body when the imaging and detection unit 10 is biased against the user's body. As shown in FIG. 2A , the imaging and detection unit 10 is shaped so that its horizontal (left-right) length is longer than its vertical (up-down) length when worn, and chest connection pads 18a and 18b are located near the left and right ends of the imaging and detection unit 10. This arrangement makes it possible to suppress left-right rotational shake during imaging with the camera body 1. Furthermore, the presence of chest connection pads 18a and 18b prevents the power switch 11 and imaging mode switch 12 from coming into contact with the body. Furthermore, chest connection pads 18a and 18b prevent heat from being transferred to the user's body even if the temperature of the imaging and detection unit 10 rises during long-term imaging, and also serve to adjust the angle of the imaging and detection unit 10.

[0054] FIG. 2D is a top view of the imaging and detection unit 10.

[0055] As shown in FIG. 2D, a face direction detection window 13 is provided in the center of the top surface of the imaging and detection unit 10, and chest connection pads 18a and 18b protrude from the imaging and detection unit 10.

[0056] FIG. 2E is a diagram showing the configuration of the face direction detection section 20, which is located inside the imaging / detection section 10 and below the face direction detection window 13.

[0057] The face direction detection unit 20 includes an infrared LED 22 and an infrared condenser lens 26. The face direction detection unit 20 further includes an infrared LED lighting circuit 21 and an infrared detection processing device 27 shown in FIG.

[0058] The infrared LED 22 projects infrared rays 23 (FIG. 5) toward the user.

[0059] The infrared condenser lens 26 is a lens that focuses reflected light rays 25 (FIG. 5) reflected from the user when the infrared LED 22 emits infrared rays 23 onto a sensor (not shown) of the infrared detection processing device 27 .

[0060] FIG. 2F is a view of camera body 1 worn by a user as seen from the left side of the user.

[0061] Angle adjustment button 85L is provided on angle holding unit 81L and is used when adjusting the angle of the imaging / detection unit 10. Although not shown in this drawing, an angle adjustment button is also provided inside angle holding unit 81R on the opposite side, in a position symmetrical to angle adjustment button 85L.

[0062] The angle adjustment button is visible in Figures 2A, 2C, and 2D, but is omitted for simplicity of explanation.

[0063] The user can change the angle between the imaging / detection unit 10 and angle holding unit 81L by pressing the angle adjustment button 85L and moving the angle holding unit 81L up or down as shown in Figure 2F. The same is true for the right side. The protruding angle of the chest connection pads 18a and 18b can also be changed. With the operation of these two types of angle change members (angle adjustment button and chest connection pad), the imaging / detection unit 10 can adjust the optical axis of the imaging lens 16 horizontally, regardless of individual differences in the user's chest position and shape.

[0064] FIG. 3 is a diagram illustrating the battery section 90 in detail.

[0065] FIG. 3(a) is a partially see-through view of the battery unit 90 from the rear.

[0066] As shown in Figure 3(a), in order to balance the weight of the battery unit 90, two batteries, a left battery 94L and a right battery 94R, are mounted symmetrically inside. By arranging the left and right batteries 94L, 94R symmetrically with respect to the center of the battery unit 90 in this way, the left and right weight balance is matched and misalignment of the camera body 1 is prevented. Note that the battery unit 90 may be configured to mount only one battery.

[0067] 3(b) is a view from above of the battery section 90. In this figure, the batteries 94L and 94R are also shown in perspective.

[0068] As shown in FIG. 3(b), by symmetrically arranging batteries 94L and 94R on both sides of the spine-protection cutout 93, the relatively heavy battery section 90 can be worn by the user without any burden.

[0069] Figure 3(c) is a view of the battery section 90 as seen from the back side, that is, as seen from the side that comes into contact with the user's body, that is, from the opposite side to Figure 3(a).

[0070] As shown in FIG. 3(c), the spine-protection notch 93 is provided in the center so as to be aligned with the user's spine.

[0071] 4 is a functional block diagram of the camera body 1. Details will be described later, so here we will use FIG. 4 to explain the general flow of processing executed by the camera body 1.

[0072] 4, the camera body 1 includes a face direction detection unit 20, a recording direction / angle of view determination unit 30, a photographing unit 40, an image cropping / development processing unit 50, a primary recording unit 60, a transmission unit 70, and other control units 111. These functional blocks are executed under the control of an overall control CPU 101 (FIG. 5) that performs overall control of the camera body 1.

[0073] The face direction detection unit 20 is a functional block executed by the infrared LED 22 and infrared detection processing device 27, etc., and detects the face direction to infer the observation direction and passes it to the recording direction and angle of view determination unit 30.

[0074] The recording direction and angle of view determination unit 30 performs various calculations based on the observation direction inferred by the face direction detection unit 20 and the calibration data described below, determines information on the position and range when cutting out the image from the shooting unit 40, and passes this information to the image cutting and development processing unit 50.

[0075] The photographing unit 40 converts light rays from the subject into a wide-angle image and passes the image to the image cutting and development processing unit 50.

[0076] The image cutting and development processing unit 50 uses information from the recording direction and angle of view determination unit 30 to cut out and develop a range centered on the user's observation direction from the image from the shooting unit 40, and passes it to the primary recording unit 60.

[0077] The primary recording unit 60 is a functional block that includes the primary memory 103 (FIG. 5) and the like, and records video information and transfers it to the transmitting unit 70 at the required timing.

[0078] The transmitter 70 is wirelessly connected to the display device 800 (FIG. 1D), the calibrator 850, and the simple display device 900, which are predetermined communication partners, and communicates with them.

[0079] The display device 800 can be connected to the transmitter 70 via a wireless LAN capable of high-speed connection (hereinafter referred to as "high-speed wireless"). In this embodiment, wireless communication compatible with the IEEE802.11ax (WiFi 6) standard is used as the high-speed wireless communication, but wireless communication compatible with other standards, such as the WiFi 4 standard or the WiFi 5 standard, may also be used. Furthermore, the display device 800 may be a device developed specifically for the camera body 1, or may be a general smartphone, tablet terminal, or the like.

[0080] The connection between the transmitter 70 and the display device 800 may be made using low-power wireless, or may be made using both high-speed wireless and low-power wireless, or may be made by switching between the two. In this embodiment, large amounts of data, such as video files of moving images (described later), are transmitted using high-speed wireless, while small amounts of data or data that may take a long time to transmit are transmitted using low-power wireless. Here, in this embodiment, Bluetooth is used as the low-power wireless, but other short-range (short-distance) wireless communication such as NFC (Near Field Communication) may also be used.

[0081] Calibrator 850 is an electronic device that performs initial settings and individual settings for camera body 1, and can be connected to transmitter 70 via high-speed wireless communication, just like display device 800. Details of calibrator 850 will be described later. Furthermore, display device 800 may also function as calibrator 850.

[0082] The simple display device 900 is a display device that can be connected to the transmitting unit 70 only by low-power wireless communication, for example.

[0083] Due to time constraints, simple display device 900 cannot transmit moving images to and from transmitter 70, but can transmit timing information for starting and stopping image capture, and can check images to the extent of confirming composition, etc. Similarly to display device 800, simple display device 900 may be a device developed specifically for camera body 1, or may be a smart watch or the like.

[0084] Figure 5 is a block diagram showing the hardware configuration of the camera body 1. Furthermore, the same reference numerals are used for the configurations and functions already explained using Figures 1A to 1C, etc., and detailed explanations will be omitted.

[0085] 5, the camera body 1 includes an overall control CPU 101, a power switch 11, an imaging mode switch 12, a face direction detection window 13, a start switch 14, a stop switch 15, an imaging lens 16, and an LED 17.

[0086] The camera body 1 also includes an infrared LED lighting circuit 21, an infrared LED 22, an infrared condenser lens 26, and an infrared detection processing device 27, which constitute a face direction detection unit 20 (FIG. 4).

[0087] The camera body 1 also includes an imaging unit 40 (FIG. 4) consisting of an imaging driver 41, a solid-state imaging element 42, and an imaging signal processing circuit 43, and a transmission unit 70 (FIG. 4) consisting of a low-power wireless unit 71 and a high-speed wireless unit 72.

[0088] Although the camera body 1 has only one imaging unit 40 in this embodiment, it may have two or more imaging units 40 to capture 3D images, capture images with a wider angle of view than can be obtained with one imaging unit 40, or capture images in multiple directions.

[0089] The camera body 1 also includes various types of memory such as a large-capacity nonvolatile memory 51, an internal nonvolatile memory 102, and a primary memory 103.

[0090] Furthermore, the camera body 1 includes an audio processing unit 104 , a speaker 105 , a vibrator 106 , an angular velocity sensor 107 , an acceleration sensor 108 , and various switches 110 .

[0091] The overall control CPU 101 is connected to the power switch 11 and other components described above with reference to Figure 2C, and controls the camera body 1. The recording direction and angle of view determination unit 30, image cropping and development processing unit 50, and other control unit 111 shown in Figure 4 are realized by the overall control CPU 101 itself.

[0092] The infrared LED lighting circuit 21 controls the turning on and off of the infrared LED 22 described above with reference to FIG. 2E, and controls the projection of infrared rays 23 from the infrared LED 22 toward the user.

[0093] The face direction detection window 13 is made up of a visible light cut filter, which does not transmit much visible light but allows infrared light 23 and its reflected light 25, which are light in the infrared range, to pass through sufficiently.

[0094] The infrared condenser lens 26 is a lens that condenses the reflected light ray 25 .

[0095] The infrared detection processing device 27 has a sensor that detects the reflected light beam 25 focused by the infrared focusing lens 26. This sensor converts the image formed by the focused reflected light beam 25 into sensor data and passes it to the overall control CPU 101.

[0096] 1B, when the user wears the camera body 1, the face direction detection window 13 is located under the user's chin. Therefore, infrared rays 23 emitted from the infrared LED 22 pass through the face direction detection window 13 and are irradiated onto an infrared irradiation surface 24 located near the user's chin, as shown in FIG. 5. Furthermore, reflected light rays 25 reflected by the infrared irradiation surface 24 pass through the face direction detection window 13 and are condensed by an infrared condensing lens 26 onto a sensor in an infrared detection processing device 27.

[0097] The various switches 110 are not shown in FIGS. 1A to 1C and are switches for executing functions that are not related to this embodiment, although details will be omitted.

[0098] The imaging driver 41 includes a timing generator and the like, generates and outputs various timing signals to each section involved in imaging, and drives the solid-state imaging element 42 .

[0099] The solid-state image pickup device 42 outputs to the image pickup signal processing circuit 43 a signal obtained by photoelectrically converting the subject image projected through the image pickup lens 16 described with reference to FIG. 1A.

[0100] The imaging signal processing circuit 43 performs processes such as clamping and A / D conversion on the signal from the solid-state imaging device 42 and outputs the generated imaging data to the overall control CPU 101 .

[0101] The built-in nonvolatile memory 102 uses a flash memory or the like and stores the startup program of the overall control CPU 101 and setting values ​​for various program modes. In this embodiment, the observation field (angle of view) and the effect level of vibration isolation control can be set, so these setting values ​​are also recorded.

[0102] The primary memory 103 is configured by RAM or the like, and temporarily stores video data being processed and the results of calculations by the overall control CPU 101.

[0103] The large-capacity nonvolatile memory 51 stores image data. In this embodiment, the large-capacity nonvolatile memory 51 is a non-removable semiconductor memory. However, the large-capacity nonvolatile memory 51 may be configured as a removable recording medium such as an SD card, or may be used in combination with the built-in nonvolatile memory 102.

[0104] The low-power wireless unit 71 exchanges data with the display device 800, the calibrator 850, and the simple display device 900 via low-power wireless communication.

[0105] The high-speed wireless unit 72 exchanges data with the display device 800 and the calibrator 850 via high-speed wireless communication.

[0106] The audio processor 104 processes external sounds (analog signals) picked up by the microphones 19L and 19R to generate audio signals.

[0107] The LED 17, speaker 105 and vibrator 106 emit light, sound and vibrate to notify or warn the user of the state of the camera body 1.

[0108] The angular velocity sensor 107 is a sensor that uses a gyro or the like, and detects the movement of the camera body 1 itself as gyro data.

[0109] The acceleration sensor 108 detects the attitude of the imaging / detection unit 10 .

[0110] Fig. 6 is a block diagram showing the hardware configuration of the display device 800. For simplicity of explanation, the same reference numerals are used for the parts explained using Fig. 1D, and explanations thereof will be omitted.

[0111] 6, the display device 800 includes a display device control unit 801, an A button 802, a display unit 803, a B button 804, a face sensor 806, an angular velocity sensor 807, an acceleration sensor 808, an imaging signal processing circuit 809, and various switches 811.

[0112] The display device 800 also includes an internal nonvolatile memory 812, a primary memory 813, a large-capacity nonvolatile memory 814, a speaker 815, a vibrator 816, an LED 817, an audio processing unit 820, a low-power wireless unit 871, and a high-speed wireless unit 872. Each of the above elements is connected to the display device control unit 801.

[0113] The display device control unit 801 is configured by a CPU and controls the display device 800 .

[0114] The imaging signal processing circuit 809 has the same functions as the imaging driver 41, solid-state imaging element 42, and imaging signal processing circuit 43 inside the camera body 1, and together with the in-camera lens 805a, constitutes the in-camera 805 shown in Fig. 1D. The data output by the imaging signal processing circuit 809 is processed within the display device control unit 801. The details of how this data is processed will be described later.

[0115] The various switches 811 are switches for executing functions that are not related to this embodiment.

[0116] The angular velocity sensor 807 is a sensor that uses a gyro or the like, and detects the movement of the display device 800 itself.

[0117] The acceleration sensor 808 detects the attitude of the display device 800 itself.

[0118] The built-in nonvolatile memory 812 uses a flash memory or the like, and stores the boot program for the display device control unit 801 and setting values ​​for various program modes.

[0119] The primary memory 813 is configured with RAM or the like, and temporarily stores video data being processed and temporarily stores the calculation results of the imaging signal processing circuit 809. In this embodiment, while a moving image is being recorded, gyro data detected by the angular velocity sensor 107 at the imaging time of each frame is associated with each frame and stored in the primary memory 813.

[0120] The large-capacity nonvolatile memory 814 stores image data of the display device 800. In this embodiment, the large-capacity nonvolatile memory 814 is configured as a removable memory such as an SD card. However, it may also be configured as a non-removable memory such as the large-capacity nonvolatile memory 51 in the camera body 1.

[0121] The speaker 815, the vibrator 816, and the LED 817 emit sound, vibration, or light to notify or warn the user of the state of the display device 800.

[0122] The audio processor 820 processes external sounds (analog signals) picked up by the left microphone 819L and the right microphone 819R to generate audio signals.

[0123] The low-power wireless unit 871 exchanges data with the camera body 1 via low-power wireless communication.

[0124] The high-speed wireless unit 872 exchanges data with the camera body 1 via high-speed wireless communication.

[0125] The face sensor 806 includes an infrared LED lighting circuit 821 , an infrared LED 822 , an infrared condenser lens 826 , and an infrared detection processing device 827 .

[0126] The infrared LED lighting circuit 821 is a circuit having the same function as the infrared LED lighting circuit 21 in FIG. 5, and controls the turning on and off of the infrared LED 822, and controls the projection of infrared rays 823 from the infrared LED 822 toward the user.

[0127] The infrared condenser lens 826 is a lens that condenses the reflected light 825 of the infrared light 823 .

[0128] The infrared detection processing device 827 has a sensor that detects the reflected light beam condensed by the infrared condensing lens 826. This sensor converts the condensed reflected light beam 825 into sensor data and passes it to the display device control unit 801.

[0129] 1D is directed toward the user, infrared rays 823 emitted from infrared LEDs 822 are irradiated onto an infrared irradiation surface 824, which is the entire face of the user, as shown in Fig. 6. Also, reflected light rays 825 reflected by the infrared irradiation surface 824 are condensed by an infrared condensing lens 826 onto a sensor in an infrared detection processing device 827.

[0130] The other function unit 830 executes smartphone functions such as a telephone function that is not related to this embodiment.

[0131] The following describes how to use the camera body 1 and the display device 800.

[0132] FIG. 7A is a flowchart showing an outline of the imaging and recording process according to this embodiment, which is executed by the camera body 1 and the display device 800.

[0133] To aid in explanation, in Fig. 7A, to the right of each step is written which device shown in Fig. 4 performs that step. That is, S100 to S700 in Fig. 7A are performed by camera body 1, and S800 to S1000 in Fig. 7A are performed by display device 800.

[0134] When the power switch 11 is turned on and the camera body 1 is powered on, the overall control CPU 101 starts up and reads a startup program from the built-in nonvolatile memory 102. Then, in S100, the overall control CPU 101 executes preparatory operation processing to set up the camera body 1 before imaging. Details of the preparatory operation processing will be described later using FIG. 7B.

[0135] In S200, the face direction detection unit 20 detects the face direction, and then executes face direction detection processing to infer the observation direction. Details of the face direction detection processing will be described later with reference to Fig. 7C. This processing is executed at a predetermined frame rate.

[0136] In S300, the recording direction and angle of view determination unit 30 executes a recording direction and range determination process. Details of the recording direction and range determination process will be described later with reference to FIG.

[0137] In S400, the imaging unit 40 captures an image and generates imaging data.

[0138] In S500, the image cutout / development processing unit 50 cuts out an image from the imaging data generated in S400 using the recording direction and angle of view information determined in S300, and executes a recording range development process to perform development processing on that range. Details of the recording range development process will be described later using FIG. 7E.

[0139] In S600, the primary recording unit 60 executes a primary recording process to store the video developed in S500 as video data in the primary memory 103. Details of the primary recording process will be described later with reference to FIG.

[0140] In S700, the transmitter 70 executes a transfer process to the display device 800, in which the video primarily recorded in S600 is wirelessly transmitted to the display device 800 at a specified timing. The details of the transfer process to the display device 800 will be described later with reference to FIG.

[0141] The steps from S800 onwards are executed by the display device 800.

[0142] In S800, the display device control unit 801 executes optical correction processing to correct optical aberrations in the image transferred in S700 from the camera body 1. Details of the optical correction processing will be described later using FIG.

[0143] In S900, the display device control unit 801 performs stabilization processing on the image that has been optically corrected in S800. Details of the stabilization processing will be described later with reference to FIG.

[0144] The order of S800 and S900 may be reversed, i.e., image stabilization processing may be performed first, followed by optical correction.

[0145] In S1000, the display device control unit 801 performs secondary recording in the large-capacity nonvolatile memory 814 of the image that has undergone the optical correction processing and image stabilization processing in S800 and S900, and then ends this processing.

[0146] Next, with reference to FIGS. 7B to 7F, the subroutines of the steps explained in FIG. 7A will be explained in detail together with the order of processing, using other figures.

[0147] Fig. 7B is a flowchart of the subroutine of the preparatory operation process of S100 in Fig. 7A. This process will be explained below with reference to the various parts shown in Figs.

[0148] In S101, it is determined whether or not the power switch 11 is ON. If the power remains OFF, the process waits, and if the power is ON, the process proceeds to S102.

[0149] In S102, it is determined which mode has been selected by the imaging mode switch 12. If the result of the determination is that the mode selected by the imaging mode switch 12 is the moving image mode, the process proceeds to S103.

[0150] In S103, various movie mode settings are read from the built-in nonvolatile memory 102 and stored in the primary memory 103, after which the process proceeds to S104. Here, the various movie mode settings include the angle of view setting value V (preset to 90° in this embodiment) and the vibration isolation level specified as "strong," "medium," "off," etc.

[0151] In S104, the operation of the imaging driver 41 for the moving image mode is started, and then the process exits this subroutine.

[0152] If it is determined in S102 that the mode selected by the imaging mode switch 12 is the still image mode, the process proceeds to S106.

[0153] In S106, various settings for the still image mode are read from the built-in nonvolatile memory 102 and stored in the primary memory 103, and then the process proceeds to S107. Here, the various settings for the still image mode include the angle of view setting value V (preset to 45° in this embodiment) and the vibration isolation level specified as "strong," "medium," "off," etc.

[0154] In S107, the operation of the imaging driver 41 for the still image mode is started, and then the process exits this subroutine.

[0155] If the result of the determination in S102 is that the mode selected by the imaging mode switch 12 is the pre-setting mode, the process proceeds to S108. Here, the pre-setting mode is a mode in which the imaging mode is set for the camera body 1 from an external device such as the display device 800, and is one of three imaging modes that can be switched by the imaging mode switch 12. The pre-setting mode is, in other words, a mode for custom photography. Here, because the camera body 1 is a small, wearable device, the camera body 1 is not provided with operation switches, setting screens, etc. for changing the detailed settings of the camera body 1, and the detailed settings of the camera body 1 are changed by an external device such as the display device 800.

[0156] For example, consider a case where you want to capture the same video at a 90° angle of view and then at a 110° angle of view. Since the angle of view is set to 90° in normal video mode, to capture such video, you must first capture the video in normal video mode, then stop video capture, display the settings screen for camera body 1 on display device 800, and switch the angle of view to 110°. However, if an event is in progress, performing such an operation on display device 800 can be cumbersome.

[0157] On the other hand, if the pre-setting mode is set in advance to a mode for capturing video at a 110° angle of view, after capturing video at a 90° angle of view, the user can instantly change to capturing video at a 110° angle of view simply by sliding the capture mode switch 12 to "Pre." In other words, the user does not need to interrupt their current activity and perform the above-mentioned troublesome operation.

[0158] In addition, the settings made in the pre-setting mode may include not only the angle of view, but also the vibration isolation level specified as "strong," "medium," or "off," as well as voice recognition settings not described in this embodiment.

[0159] In S108, various settings of the pre-setting mode are read from the built-in non-volatile memory 102 and stored in the primary memory 103, and then the process proceeds to S109. Here, the various settings of the pre-setting mode include the angle of view setting value V and the image stabilization level specified as "strong," "medium," "off," etc.

[0160] In S109, the operation of the imaging driver 41 for the pre-setting mode is started, and then the process exits from this subroutine.

[0161] Here, various settings of the video mode read out in S103 will be described with reference to FIG.

[0162] 13 is a diagram showing a menu screen for various settings in video mode that is displayed on display unit 803 of display device 800 before capturing an image with camera body 1. Note that the same reference numerals are used for the same parts as in FIG. 1D and their explanations will be omitted. Note that display unit 803 has a touch panel function, and the following explanation will be given assuming that it functions by touch operations, including operations such as swiping.

[0163] 13, the menu screen includes a preview screen 831, a zoom lever 832, a recording start / stop button 833, a switch 834, a battery level indicator 835, a button 836, a lever 837, and an icon display section 838.

[0164] On the preview screen 831, the image captured by the camera body 1 can be checked, and the zoom amount and angle of view can be confirmed.

[0165] The zoom lever 832 is an operation unit that can set the zoom by shifting it left or right. In this embodiment, four values ​​of 45°, 90°, 110°, and 130° can be set as the angle of view setting value V, but the zoom lever 832 may be configured to set values ​​other than these as the angle of view setting value V.

[0166] The recording start / stop button 833 is a toggle switch that combines the functions of the start switch 14 and the stop switch 15 .

[0167] The switch 834 is a switch for switching the vibration isolation between "off" and "on."

[0168] The remaining battery power display 835 displays the remaining battery power of the camera body 1.

[0169] Button 836 is a button for changing the mode.

[0170] Lever 837 is a lever for setting the vibration isolation level. In this embodiment, only "strong" and "medium" can be set as the vibration isolation level, but other vibration isolation levels, such as "weak," may also be set. Furthermore, the vibration isolation level may be set in a stepless manner.

[0171] The icon display section 838 displays a plurality of thumbnail icons for preview.

[0172] Fig. 7C is a flowchart of the subroutine of the face direction detection process of S200 in Fig. 7A. Before describing the details of this process, a method of detecting the face direction by infrared projection will be described with reference to Figs. 8A to 8K.

[0173] FIG. 8A is a diagram showing a visible light image of the user's face as seen from the position of face direction detection window 13. FIG.

[0174] The image in FIG. 8A is identical to the image captured by the visible light imaging element when the face direction detection window 13 does not have a visible light cut filter component, the visible light is sufficiently transmitted, and the infrared detection processing device 27 is a visible light imaging element.

[0175] The image in FIG. 8A shows the front of the neck 201, the base of the chin 202, the tip of the chin 203, and the face 204 including the nose when imaging from the upper part of the user's sternum.

[0176] FIG. 8B is a diagram showing a case where a fluorescent lamp 205 in the room is reflected as a background in the visible light image of the user shown in FIG. 8A.

[0177] The visible light image in Figure 8B shows multiple fluorescent lights 205 around the user. As such, various backgrounds and the like are reflected in the image of the user depending on the conditions of use, making it difficult for the face direction detection unit 20 and overall control CPU 101 to separate the image of the face from the visible light image. On the other hand, there is technology for separating such images using AI, etc., but this requires high performance from the overall control CPU 101 and is not suitable for the camera body 1, which is a portable device.

[0178] Therefore, the camera 1 of the first embodiment detects the user's face using an infrared light image. The face direction detection window 13 is configured with a visible light cut filter, which does not transmit much visible light, so the image of the infrared detection processing device 27 will not be like the images shown in Figures 8A and 8B.

[0179] FIG. 8C shows an infrared light image of the user shown in FIG. 8B and the fluorescent light in the background when the sensor of the infrared detection processing device 27 forms an image through the face direction detection window 13 without turning on the infrared LED 22.

[0180] 8C, the user's neck and chin appear dark, while the fluorescent light 205 appears slightly brighter because it contains not only visible light but also infrared light components.

[0181] FIG. 8D shows an image of the user shown in FIG. 8B and a fluorescent lamp in the background, which is formed by the sensor of the infrared detection processing device 27 through the face direction detection window 13 with the infrared LED 22 turned on.

[0182] In the image of Fig. 8D, the user's neck and chin are brighter, whereas, unlike Fig. 8C, the brightness around the fluorescent lamp 205 remains unchanged.

[0183] Fig. 8E is a diagram showing a difference image calculated by subtracting the image of Fig. 8C from the image of Fig. 8D, and it can be seen that the user's face appears.

[0184] In this way, the overall control CPU 101 calculates the difference between the images formed by the sensor of the infrared detection processing device 27 when the infrared LED 22 is on and when it is off, thereby obtaining a differential image (hereinafter also referred to as a facial image) in which the user's face is extracted.

[0185] The face direction detection unit 20 of this embodiment employs a method of acquiring a face image by extracting infrared reflection intensity as a two-dimensional image using an infrared detection processing device 27. The sensor of the infrared detection processing device 27 employs a structure similar to that of a general image sensor, and acquires a face image one frame at a time. A vertical synchronization signal (hereinafter referred to as a V signal) for frame synchronization is generated by the infrared detection processing device 27 and output to the overall control CPU 101.

[0186] FIG. 9 is a timing chart showing the timing of turning on and off the infrared LED 22 and related signals.

[0187] 9(a) shows the timing at which the V signal is generated by the infrared detection processing device 27. When the V signal goes high, frame synchronization and the timing at which the infrared LED 22 is turned on and off are determined.

[0188] In Fig. 9(a), t1 indicates the first facial image acquisition period, and t2 indicates the second facial image acquisition period. Figs. 9(a), (b), (c), and (d) are drawn so that the horizontal time axes are the same.

[0189] In Figure 9(b), the vertical axis represents the H position of the image signal output from the sensor of the infrared detection processing device 27. The infrared detection processing device 27 controls the movement of its sensor so that the H position of the image signal is synchronized with the V signal, as shown in Figure 9(b). As mentioned above, the sensor of the infrared detection processing device 27 has the same structure as a general image sensor, and its movement is well known, so detailed control will not be described here.

[0190] 9(c) shows the timing of switching between Hi and Low of the IR-ON signal output from the overall control CPU 101 to the infrared LED lighting circuit 21. The overall control CPU 101 controls the switching between Hi and Low of the IR-ON signal so as to be synchronized with the V signal, as shown in FIG. 9(c). Specifically, the overall control CPU 101 outputs a Low IR-ON signal to the infrared LED lighting circuit 21 during a period t1, and outputs a Hi IR-ON signal to the infrared LED lighting circuit 21 during a period t2.

[0191] Here, while the IR-ON signal is Hi, the infrared LED lighting circuit 21 lights up the infrared LED 22, and infrared rays 23 are projected toward the user. On the other hand, while the IR-ON signal is Low, the infrared LED lighting circuit 21 turns off the infrared LED 22.

[0192] FIG. 9(d) shows imaging data output from the sensor of the infrared detection processing device 27 to the overall control CPU 101. The vertical direction represents signal intensity, which indicates the amount of reflected light ray 25 received. That is, during period t1, the infrared LED 22 is off, so there is no reflected light ray 25 from the user's face, and imaging data such as that shown in FIG. 8C is obtained. On the other hand, during period t2, the infrared LED 22 is on, so there is reflected light ray 25 from the user's face, and imaging data such as that shown in FIG. 8D is obtained. Therefore, as shown in FIG. 9(d), the signal intensity during period t2 is higher than that during period t1 by the amount of reflected light ray 25 from the user's face.

[0193] Figure 9(e) shows the result of subtracting the imaging data during the period t1 from the imaging data during the period t2 in Figure 9(d), resulting in facial image data in which only the component of the reflected light ray 25 from the user's face is extracted, as shown in Figure 8E.

[0194] FIG. 7C shows the face direction detection process in S200, including the operations described above with reference to FIGS. 8C to 8E and 9.

[0195] First, in S201, the face direction detection unit 20 acquires timing V1 at which the period t1 starts when the V signal output from the infrared detection processing device 27 becomes Hi. Once timing V1 is acquired, the process proceeds to S202.

[0196] In S202, the face direction detection unit 20 sets the IR-ON signal to Low and outputs it to the infrared LED lighting circuit 21. As a result, the infrared LED 22 is turned off.

[0197] In S203, the face direction detection unit 20 reads one frame of image pickup data output from the infrared detection processing device 27 during the period t1, and temporarily stores the data in the primary memory 103 as Frame1.

[0198] In S204, when the V signal output from the infrared detection processing device 27 reaches timing V2 at which the period t2 starts, the face direction detection unit 20 proceeds to S205.

[0199] In S205, the face direction detection unit 20 sets the IR-ON signal to Hi and outputs it to the infrared LED lighting circuit 21. As a result, the infrared LED 22 lights up.

[0200] In S206, the face direction detection unit 20 reads one frame of image pickup data output from the infrared detection processing device 27 during the period t2, and temporarily stores the data in the primary memory 103 as Frame2.

[0201] In S207, the face direction detection unit 20 sets the IR-ON signal to Low and outputs it to the infrared LED lighting circuit 21. As a result, the infrared LED 22 is turned off.

[0202] In S208, the face direction detection unit 20 reads Frame1 and Frame2 from the primary memory 103. Then, the face direction detection unit 20 subtracts Frame1 from Frame2 to calculate the light intensity Fn of the reflected light ray 25 component of the user corresponding to the face image in FIG. 9(e) (this is a process generally called black subtraction).

[0203] In S209, the face direction detection unit 20 extracts the neck position (center of neck rotation) from the light intensity Fn.

[0204] First, the overall control CPU 101 divides the face image into a plurality of distance areas, which will be explained using FIG. 8F, based on the light intensity Fn.

[0205] Figure 8F shows the case where the shading of the differential image in Figure 8E is adjusted to match the scale of the light intensity of the reflected light rays 25 of the infrared rays 23 projected onto the user's face and neck in order to see the distribution of light intensity for each part of the user's face and neck.

[0206] 8F(A) is a diagram showing the distribution of light intensity of reflected light ray 25 in the facial image of FIG. 8E, divided into regions and shown in gray scales. For the purpose of explanation, the Xf axis is taken in the direction from the center of the user's neck to the tip of the chin.

[0207] In Figure 8F(b), the horizontal axis represents the light intensity on the Xf axis in Figure 8F(a), and the vertical axis represents the Xf axis. The horizontal axis represents light intensity that increases toward the right.

[0208] In FIG. 8F(a), the face image is divided into six areas (distance areas) 211 to 216 according to light intensity.

[0209] Area 211 is the area with the most intense light intensity and is shown in white as a shade of grey.

[0210] Region 212 is a region where the light intensity is slightly lower than region 211 and is shown in a much lighter gray color as a gray scale.

[0211] Region 213 is an area where the light intensity is even lower than region 212 and is shown in light gray as a gray gradation.

[0212] Region 214 is an area where the light intensity is even lower than region 213 and is shown in a medium gray color as a gray scale.

[0213] Region 215 is a region where the light intensity is even lower than region 214, and is shown in a slightly darker gray as a gray scale.

[0214] The area 216 is the area with the weakest light intensity, and is the darkest gray in terms of gray levels. The area above the area 216 is black, with no light intensity.

[0215] This light intensity will be explained in detail below with reference to FIG.

[0216] FIG. 10 is a diagram for explaining the vertical movement of the user's face, showing the state observed from the left lateral direction of the user.

[0217] FIG. 10(a) shows a user facing forward. The imaging and detection unit 10 is located near the top of the user's sternum. Furthermore, infrared light 23 from an infrared LED 22 is projected onto the lower part of the user's head through a face direction detection window 13 at the top of the imaging and detection unit 10. The distance from the face direction detection window 13 to the neck 200 in the direction extending from the top of the user's sternum is defined as Dn, the distance from the face direction detection window 13 to the base of the chin 202 is defined as Db, and the distance from the face direction detection window 13 to the tip of the chin 203 is defined as Dc. In this case, the distances increase in the order Dn, Db, and Dc. Because light intensity is inversely proportional to the square of the distance, the light intensity of the reflected light ray 25 from the infrared irradiation surface 24 when focused on the sensor of the infrared detection processing device 27 decreases in the order of the neck 200, the base of the chin 202, and the tip of the chin 203. It can also be seen that the light intensity becomes even darker for face 204 including the nose, which is located at a distance farther than Dc from face direction detection window 13. That is, in the case of Fig. 10(a), it can be seen that an image having the light intensity distribution shown in Fig. 8F is acquired.

[0218] Note that the configuration of the face direction detection unit 20 is not limited to that shown in this embodiment as long as it can detect the face direction of the user. For example, an infrared pattern may be emitted from the infrared LED 22, and the infrared pattern reflected from the irradiated object may be detected by a sensor of the infrared detection processing device 27. In this case, the sensor of the infrared detection processing device 27 is preferably a structured light sensor. Furthermore, the sensor of the infrared detection processing device 27 may be a sensor that performs phase comparison between the infrared rays 23 and the reflected light beam 25, such as a Time of Day (ToF) sensor.

[0219] Next, extraction of the neck position in S209 of FIG. 7C will be described with reference to FIG. 8G.

[0220] FIG. 8G(a) is a diagram in which the symbols indicating each part of the user's body in FIG. 10(a), as well as the symbols of a double circle indicating the neck position and a black circle indicating the chin position, are superimposed on FIG. 8F.

[0221] The white area 211 corresponds to the neck 200 (FIG. 10(a)), the fairly light gray area 212 corresponds to the front of the neck 201 (FIG. 10(a)), the light gray area 213 corresponds to the base of the chin 202 (FIG. 10(a)), the medium gray area 214 corresponds to the tip of the chin 203 (FIG. 10(a)), and the slightly darker gray area 215 corresponds to the lips located at the bottom of the face 204 (FIG. 10(a)) and the surrounding area of ​​the lower part of the face. Furthermore, the darker gray area 216 corresponds to the nose located in the center of the face 204 (FIG. 10(a)) and the surrounding area of ​​the upper part of the face.

[0222] As shown in FIG. 10(a), the difference between the distances Db and Dc is smaller than the distance from the face direction detection window 13 to other parts of the user, and therefore the difference in the reflected light intensity between the light gray area 213 and the medium gray area 214 is also small.

[0223] On the other hand, as shown in Figure 10(a), of the distances from the face direction detection window 13 to each part of the user, the distance Dn is the shortest and closest distance, so the white area 211 corresponding to the neck 200 is the area with the strongest reflection intensity.

[0224] Therefore, the overall control CPU 101 sets the position 206 indicated by a double circle in Fig. 8G(A), which is the area 211 around the neck 200, the center of the left and right sides of the area 211, and is closest to the imaging and detection unit 10, as the position of the neck rotation center (hereinafter referred to as neck position 206). The processing up to this point is the content performed in S209 in Fig. 7C.

[0225] Next, the extraction of the chin position in S210 of FIG. 7C will be described with reference to FIG. 8G.

[0226] As shown in FIG. 8G(A), a medium-gray area 214, which is lighter than an area 215 of face 204 corresponding to the lower part of the face including the lips, is the area including the chin. As can be seen from FIG. 8G(B), the rate of change of distance from face direction detection window 13 increases, so the light intensity drops sharply in area 215 adjacent to area 214. Overall control CPU 101 determines that brighter area 214 adjacent to area 215 where the light intensity drops sharply is the chin area. Furthermore, overall control CPU 101 calculates (extracts) the position that is the horizontal center of area 214 and farthest from neck position 206 (the position indicated by a black circle in FIG. 8G(A)) as chin position 207.

[0227] For example, Figures 8H and 8I show the changes when the face is turned to the right.

[0228] Fig. 8H is a diagram showing a difference image calculated in the same way as Fig. 8E when the user's face is facing rightward. Fig. 8I is a diagram in which the shading of the difference image of Fig. 8H is adjusted to match the scale with the light intensity of the reflected infrared light beam projected onto the user's face and neck, and the double circle indicating neck position 206, which is the position of the neck rotation center, and the black circle indicating chin position 207r are superimposed.

[0229] 8I, which is located to the left when viewed from the side of the imaging and detection unit 10. Region 215, which corresponds to the lower part of face 204 including the lips, also moves to region 215r, which is located to the left when viewed from the side of the imaging and detection unit 10.

[0230] Therefore, the overall control CPU 101 determines that the region 214r in front of 215r where the light intensity drops sharply is the chin region. Furthermore, the overall control CPU 101 calculates (extracts) the position in the left-right center of 214r and farthest from the neck position 206 (the position indicated by the black circle in FIG. 8I) as the chin position 207r.

[0231] After that, the overall control CPU 101 calculates a movement angle θr that indicates how far the chin position 207r in Fig. 8I has moved rightward from the chin position 207 in Fig. 8G(A) around the neck position 206. As shown in Fig. 8I, the movement angle θr is the angle in the left-right direction of the user's face.

[0232] Using the above method, in S210, the angle of the user's face in the left-right direction is calculated from the position of the tip of the chin detected by the infrared detection processing device 27 of the face direction detection unit 20 (three-dimensional detection sensor).

[0233] Next, detection of an upward-facing face will be described.

[0234] FIG. 10(b) is a diagram showing the user's face facing horizontally, and FIG. 10(c) is a diagram showing the user's face facing upward at an angle of 33° above the horizontal.

[0235] In FIG. 10(b), the distance from the face direction detection window 13 to the chin tip 203 is Ffh, and in FIG. 10(c), the distance from the face direction detection window 13 to the chin tip 203u is Ffu.

[0236] As shown in FIG. 10(c), the chin 203u also moves up together with the face, so it can be seen that the distance Ffu is longer than that of Ffh.

[0237] FIG. 8J shows an image of the user seen through the face direction detection window 13 when the user is tilting their face 33° above horizontal. As shown in FIG. 10(c), because the user is facing upward, the face direction detection window 13, positioned under the user's chin, does not show the face 204, including the lips and nose, but only the chin 203. FIG. 8K shows the distribution of the light intensity of the reflected light ray 25 when infrared light 23 is irradiated on the user in the state shown in FIG. 10(c). FIG. 8K(A) shows the difference image calculated in the same manner as FIG. 8E, scaled to match the light intensity of the reflected light ray of the infrared light projected on the user's face and neck, with a double circle indicating the neck position 206 and a black circle indicating the chin position 207u superimposed. The graphs in FIG. 8K(B) and (C) show the density changes in the image on the left, with (B) corresponding to the graph in FIG. 8F and (C) corresponding to the graph in FIG. 8G.

[0238] The six regions 211u to 216u corresponding to light intensity in FIG. 8K(A) are regions with the same light intensity as the regions shown in FIG. 8F, denoted by the letter "u." The light intensity of the user's chin 203 was in the medium gray region 214 in FIG. 8F(A), but in FIG. 8K(A) it shifts to the gray side, to the slightly darker gray region 215u. As shown in FIG. 10(c), because Ffu is longer than Ffh, the infrared detection processing device 27 can detect that the light intensity of the reflected light beam 25 from the user's chin 203 weakens in inverse proportion to the square of the distance.

[0239] Next, detection of a face facing downward will be described.

[0240] FIG. 10(d) is a diagram showing a state in which the user is facing downward at an angle of 22 degrees from the horizontal.

[0241] In FIG. 10(d), the distance from the face direction detection window 13 to the tip of the chin 203d is Ffd.

[0242] As shown in FIG. 10(d), since the chin 203d also moves downward together with the face, the distance Ffd becomes shorter than the distance Ffh, and it can be seen that the light intensity of the reflected light ray 25 at the chin 203 becomes stronger.

[0243] Returning to FIG. 7C, in S211, the overall control CPU 101 calculates the distance from the chin position to the face direction detection window 13 based on the light intensity at the chin position detected by the infrared detection processing device 27 of the face direction detection unit 20 (three-dimensional detection sensor). Based on this, the angle of the face in the up-down direction is also calculated.

[0244] In S212, the overall control CPU 101 stores the angles of the left-right direction (first detection direction) and the vertical direction (second detection direction) perpendicular to the left-right direction of the face acquired in S210 and S211, respectively, in the primary memory 103 as the user's three-dimensional observation direction vi (where i is an arbitrary symbol). For example, when the user is observing the center of the front, the observation direction vo has a left-right direction θh of 0° and a vertical direction θv of 0°, resulting in vector information of [0°, 0°]. Similarly, when the user is observing 45° to the right, the observation direction vr has vector information of [45°, 0°].

[0245] In S211, the vertical angle of the face is calculated by detecting the distance from the face direction detection window 13, but this method is not limited to this. For example, the angle change may be calculated by comparing the variation level of the light intensity of the chin 203. That is, the angle change of the face may be calculated by comparing the gradient CDh of the reflected light intensity from the base of the chin 202 to the tip of the chin 203 in FIG. 8G(A) with the gradient CDu of the reflected light intensity from the base of the chin 202 to the tip of the chin 203 in FIG. 8K(C).

[0246] Fig. 7D is a flowchart of the subroutine for the recording direction and range determination process of S300 in Fig. 7A. Before explaining the details of this process, we will first use Fig. 11A to explain the ultra-wide-angle image for which the recording direction and recording range are determined in this embodiment.

[0247] In the camera body 1 of this embodiment, the imaging unit 40 captures an ultra-wide-angle image around the imaging / detection unit 10 using the ultra-wide-angle imaging lens 16, and by cutting out a portion of that image, an image in the observation direction can be obtained.

[0248] FIG. 11A is a diagram showing a target field of view 125 in a super-wide-angle image captured by the image capturing unit 40 when the user faces forward.

[0249] 11A, the imageable pixel area 121 of the solid-state imaging element 42 is a rectangular area. The effective projection area 122 (predetermined area) is a circular area of ​​a semi-spherical image that is fisheye projected onto the solid-state imaging element 42 by the imaging lens 16. The imaging lens 16 is adjusted so that the center of the pixel area 121 and the center of the effective projection area 122 coincide with each other.

[0250] The outermost periphery of the circular effective projection area 122 indicates a position with a field of view (FOV) angle of 180°. When the user's observation (line of sight) direction coincides with the center of the effective projection area 122, the angular range of the target field of view 125, which is the area to be captured and recorded, is 90° (half the VOF angle) centered on the center of the effective projection area 122. Note that the imaging lens 16 of this embodiment can also introduce light rays outside the effective projection area 122, and can project light rays up to a maximum FOV angle of approximately 192° onto the solid-state imaging element 42 in a fisheye manner. However, beyond the effective projection area 122, optical performance is significantly degraded, including extremely reduced resolution, reduced light intensity, and increased distortion. Therefore, in this embodiment, the recording area will be described as an example in which only the image in the observation direction is cut out from the image projected onto the pixel area 121 (hereinafter simply referred to as ultra-wide-angle image) of the hemispherical image displayed on the effective projection area 122.

[0251] In this embodiment, the vertical size of the effective projection area 122 is larger than the size of the short side of the pixel area 121, so the images at the top and bottom ends of the effective projection area 122 are outside the pixel area 121, but this is not limiting. For example, the optical system may be designed so that the entire effective projection area 122 fits within the pixel area 121 by changing the configuration of the imaging lens 16.

[0252] The invalid pixel area 123 is a pixel area of ​​the pixel area 121 that is not included in the effective projection area 122 .

[0253] The target field of view 125 is an area indicating the range within which an image in the user's observation direction is cut out from the ultra-wide-angle image, and is defined by a preset left-right, top-bottom field of view angle (here, 45°, FOV angle 90°) centered on the observation direction. In the example of Fig. 11A, the user's observation direction coincides with the center of the effective projection area 122, so the center of the target field of view 125 coincides with the center of the effective projection area 122. The observation direction in this case is designated as vo.

[0254] The super-wide-angle image shown in FIG. 11A includes subject A 131, who is a child, subject B 132, which is a staircase that the child, subject A, is trying to climb, and subject C 133, which is a playground equipment in the shape of a locomotive.

[0255] Next, Fig. 7D shows the recording direction and range determination process in S300, which is executed to obtain an image in the observation direction from the ultra-wide-angle image described above with reference to Fig. 11A. This process will be described below with reference to Figs. 12A to 12G, which show specific examples of target field of view 125.

[0256] In S301, a preset view angle setting value V is read from the primary memory 103 and acquired.

[0257] In this embodiment, all of the angles of view, 45°, 90°, 110°, and 130°, at which an image in the observation direction can be cut out from an ultra-wide-angle image by the image cutout / development processing unit 50 are stored in the built-in non-volatile memory 102 as the angle of view setting value V. Also, in any of steps S103, S106, and S108 of FIG. 7B , the angle of view setting value V included in the various setting values ​​read out from the built-in non-volatile memory 102 is set and stored in the primary memory 103.

[0258] In addition, in S301, the observation direction vi determined in S212 is determined as the recording direction, and an image of the target field of view 125 cut out from the ultra-wide-angle image with the acquired field of view setting value V centered on this direction is saved in the primary memory 103.

[0259] For example, if the field of view setting value V is 90° and the face direction detection process (FIG. 7C) detects an observation direction vo (vector information [0°, 0°]), the target field of view 125 (FIG. 11A) is set, with a horizontal and vertical angular range of 90° centered on the center O of the effective projection area 122. FIG. 11B is a diagram showing an image of the target field of view 125 cut out from the ultra-wide-angle image of FIG. 11A. In other words, the overall control CPU 101 sets the angle of the face direction detected by the face direction detection unit 20 to the observation direction vi, which is vector information indicating the relative position of the target field of view 125 with respect to the ultra-wide-angle image.

[0260] Here, in the case of observation direction vo, the influence of optical distortion caused by imaging lens 16 can be almost ignored, so the shape of the set target field of view 125 becomes the shape of target field of view 125o (FIG. 12A) after distortion conversion in S303 (described later). Hereinafter, the target field of view 125 after distortion conversion in the case of observation direction vi will be referred to as target field of view 125i.

[0261] Next, in S302, a preset vibration isolation level is read from the primary memory 103 and acquired.

[0262] In this embodiment, as described above, the image stabilization level included in the various setting values ​​read from the built-in nonvolatile memory 102 in any one of steps S103, S106, and S108 is set and saved in the primary memory 103.

[0263] In addition, in S302, the number of spare pixels for image stabilization Pis is set based on the image stabilization level obtained above.

[0264] In the image stabilization process, an image that follows the image in the direction opposite to the direction of the shake is acquired in accordance with the amount of shake of the imaging / detection unit 10. For this reason, in this embodiment, a spare image stabilization area required for image stabilization is provided around the target field of view 125i.

[0265] Furthermore, in this embodiment, a table that holds the value of the number of spare vibration compensation pixels Pis associated with each vibration compensation level is stored in the built-in nonvolatile memory 102. For example, when the vibration compensation level is "medium," a spare vibration compensation area having a width of 100 pixels, which is the number of spare vibration compensation pixels Pis read from the table, is set to surround the target field of view.

[0266] Fig. 12E is a diagram showing an example in which a spare image stabilization area corresponding to a predetermined image stabilization level is added around the target field of view 125o shown in Fig. 12A. Here, the case where the image stabilization level is "medium," that is, the number of spare image stabilization pixels Pis is 100, is described.

[0267] As shown by the dotted lines in FIG. 12E, a spare image stabilization pixel frame 126o having a width of 100 pixels, which is the number of spare image stabilization pixels Pis, is set on the top, bottom, left, and right of the target field of view 125o.

[0268] 12A and 12E, for simplicity of explanation, the case where the observation direction vi coincides with the center O (the optical axis center of the imaging lens 16) of the effective projection area 122 has been described. On the other hand, if the observation direction vi is in the peripheral area of ​​the effective projection area 122, a conversion is required to reduce the effects of optical distortion.

[0269] In S303, the shape of the target field of view 125 set in S301 is corrected (distortion converted) to generate the target field of view 125i, taking into account the observation direction vi and the optical characteristics of the imaging lens 16. Similarly, the number of spare pixels Pis for image stabilization set in S302 is also corrected, taking into account the observation direction vi and the optical characteristics of the imaging lens 16.

[0270] For example, suppose the angle of view setting value V is 90° and the user is observing 45° to the right of the center o. In this case, the observation direction vr (vector information [45°, 0°]) is determined in S212, and the range of 45° left and right and 45° up and down with the observation direction vr as the center becomes the target field of view 125. However, taking into account the optical characteristics of the imaging lens 16, the target field of view 125 is corrected to the target field of view 125r shown in FIG. 12B.

[0271] 12B, the target field of view 125r becomes wider as it approaches the periphery of the effective projection area 122, and the position of the observation direction vr is also slightly inside from the center of the target field of view 125r. This is because, in this embodiment, the imaging lens 16 has an optical design similar to that of a stereoscopic projection fisheye. Note that this relationship will change if the imaging lens 16 is designed as an equidistant projection fisheye, an equal solid angle projection fisheye, an orthogonal projection fisheye, or the like, and therefore the target field of view 125 is corrected to suit the optical characteristics.

[0272] FIG. 12F is a diagram showing an example in which a spare image stabilization area 126r corresponding to the same "medium" image stabilization level as the spare image stabilization area in FIG. 12E is provided around the target field of view 125r shown in FIG. 12B.

[0273] In the image stabilization spare area 126o (FIG. 12E), a width of 100 pixels is set on each of the top, bottom, left, and right sides of the target field of view 125o, which is the number of image stabilization spare pixels Pis. In contrast, in the image stabilization spare area 126r (FIG. 12F), the number of image stabilization spare pixels Pis is corrected and increases as the area approaches the periphery of the effective projection area 122.

[0274] As shown in Fig. 12F, the shape of the target field of view 125r and the shape of the extra image stabilization area required for image stabilization provided around it also have a larger correction amount as they approach the periphery of the effective projection area 122, as shown by extra image stabilization area 126r. This is also because, in this embodiment, the imaging lens 16 has an optical design similar to that of a stereoscopic projection fisheye lens. Note that this relationship will change if the imaging lens 16 is designed as an equidistant projection fisheye lens, an equal solid angle projection fisheye lens, an orthogonal projection fisheye lens, or the like, and therefore the extra image stabilization area 126r is corrected in accordance with the optical characteristics.

[0275] The process executed in S303, which sequentially switches the shape of the target field of view 125 and its preliminary image stabilization area in consideration of the optical characteristics of the imaging lens 16, is complex. For this reason, in this embodiment, the process of S303 is executed using a table that stores the target field of view 125i and the shape of its preliminary image stabilization area for each observation direction vi, stored in the internal nonvolatile memory 102. Note that, depending on the optical design of the imaging lens 16 mentioned above, an arithmetic formula may be stored in the overall control CPU 101, and the optical distortion value may be calculated using that arithmetic formula.

[0276] In S304, the position and size of the video recording frame are calculated.

[0277] As described above, the extra image stabilization area 126i required for image stabilization is provided around the target field of view 125i. However, when the position of the observation direction vi approaches the periphery of the effective projection area 122, the shape of the extra image stabilization area 126i becomes quite special, for example, like the extra image stabilization area 126r.

[0278] The overall control CPU 101 can cut out and develop only the image of such a specially shaped range. However, it is not common to use non-rectangular image when recording it as image data in S600 or transferring it to the display device 800 in S700. Therefore, in S304, the position and size of a rectangular image recording frame 127i that encompasses the entire image stabilization spare area 126i is calculated.

[0279] FIG. 12F shows an image recording frame 127r, indicated by a dashed line, calculated in S304 for the image stabilization preliminary region 126r.

[0280] In S305, the position and size of the video recording frame 127i calculated in S304 are recorded in the primary memory 103.

[0281] In this embodiment, the coordinates Xi, Yi of the upper left corner of the video recording frame 127i in the ultra-wide-angle image are recorded as the position of the video recording frame 127i, and the width WXi and height WYi of the video recording frame 127i from the coordinates Xi, Yi are recorded as the size of the video recording frame 127i. For example, for the video recording frame 127r shown in FIG. 12F, the coordinates Xr, Yr, width WXr, and height WYr shown in the figure are recorded in S305. The coordinates Xi, Yi are XY coordinates with a predetermined reference point, specifically, the optical center of the imaging lens 16, as the origin.

[0282] Once the image stabilization spare area 126i and the video recording frame 127i have been determined in this manner, the subroutine shown in FIG. 7D is exited.

[0283] In the explanation up to this point, in order to simplify the explanation of complex optical distortion conversion, examples of the observation direction vi have been given using observation directions that include horizontal 0°, i.e., observation direction vo (vector information [0°, 0°]) and observation direction vr (vector information [45°, 0°]). However, in reality, the user's observation direction vi can be in a variety of directions. Therefore, the following will explain the recording range development process that is executed when the observation direction is not horizontal 0°.

[0284] For example, when the angle of view setting value V is 90° and the observation direction vm is [−42°, −40°], the target field of view of 125 m is as shown in FIG. 12C.

[0285] Furthermore, even if the observation direction vm (vector information [-42°, -40°]) is the same as the target field of view 125m, when the field of view setting value V is 45°, the target field of view becomes 128m, which is slightly smaller than the target field of view 125m, as shown in Fig. 12D. Furthermore, for the target field of view 128m, a spare image stabilization area 129m and a video recording frame 130m are set, as shown in Fig. 12G.

[0286] S400 is a basic operation for imaging, and a detailed description thereof will be omitted since it uses a general sequence of the imaging unit 40. In this embodiment, the imaging signal processing circuit 43 in the imaging unit 40 also performs processing to correct the signal in a specific output format (examples of standards: MIPI, SLVS) output from the solid-state imaging element 42 into imaging data of a general sensor readout method.

[0287] When the mode selected by the imaging mode switch 12 is the moving image mode, the imaging unit 40 starts recording in response to the pressing of the start switch 14. Thereafter, the recording ends when the stop switch 15 is pressed. On the other hand, when the mode selected by the imaging mode switch 12 is the still image mode, the imaging unit 40 captures a still image every time the start switch 14 is pressed.

[0288] FIG. 7E is a flowchart of the subroutine for the recording area development process in S500 of FIG. 7A.

[0289] In S501, Raw data of the entire area of ​​the imaging data (ultra-wide-angle image) generated by the imaging unit 40 in S400 is acquired and input to an image capture unit called a head unit (not shown) of the overall control CPU 101.

[0290] Next, in S502, a portion of the video recording frame 127i is cropped from the ultra-wide-angle image acquired in S501 based on the coordinates Xi, Yi, width WXi, and height WYi recorded in primary memory 103 in S305. After this cropping, crop development processing (FIG. 7F) consisting of steps S503 to S508 is started, which is performed only on the pixels within the image stabilization spare area 126i. This allows for a significant reduction in the amount of calculations compared to when development processing is performed on the entire area of ​​the ultra-wide-angle image read in S501, thereby reducing calculation time and power consumption.

[0291] 7F, when the mode selected by the imaging mode switch 12 is the video mode, the processes of S200 and S300 and the process of S400 are executed in parallel at the same or different frame rates. That is, every time Raw data of the entire area for one frame generated by the imaging unit 40 is acquired, crop development processing is performed based on the coordinates Xi, Yi, width WXi, and height WYi recorded in the primary memory 103 at that time.

[0292] When the crop development process for the pixels in the shake compensation preliminary area 126i is started, first, in S503, color interpolation is performed to interpolate color pixel information arranged in a Bayer array.

[0293] Thereafter, white balance adjustment is performed in S504, and color conversion is performed in S505.

[0294] In S506, gamma correction is performed to correct the gradation in accordance with a preset gamma correction value.

[0295] In S507, edge enhancement is performed in accordance with the image size.

[0296] In S508, the data is converted into a data format that can be stored temporarily by performing compression or other processing, and is recorded in the primary memory 103, after which the subroutine is terminated. Details of this data format that can be stored temporarily will be described later.

[0297] The order of the crop development processes executed in S503 to S508 and whether or not the processes are performed may be determined in accordance with the camera system, and do not limit the present invention.

[0298] Furthermore, if the moving image mode is selected, the processes from S200 to S500 are repeatedly executed until the recording is completed.

[0299] This process can significantly reduce the amount of calculation compared to when development processing is performed on the entire area read in S501. This allows an inexpensive, low-power microcomputer to be used as the overall control CPU 101, reduces heat generation in the overall control CPU 101, and improves battery life.

[0300] Furthermore, in this embodiment, in order to reduce the control load of the overall control CPU 101, the optical correction processing (S800 in FIG. 7A) and image stabilization processing (S900 in FIG. 7A) of the image are not performed by the camera body 1, but are transferred to the display device 800 and then performed by the display device control unit 801. Therefore, if only data of an image partially cropped from the projected ultra-wide-angle image is sent to the display device 800, the optical correction processing and image stabilization processing cannot be performed. In other words, the cropped image data alone does not contain position information used to substitute into an equation for optical correction processing or to refer to a correction table for image stabilization processing, and therefore these processes cannot be correctly performed on the display device 800. For this reason, in this embodiment, not only the data of the cropped image but also correction data including information such as the crop position of the image from the ultra-wide-angle image is transmitted from the camera body 1 to the display device 800.

[0301] If the extracted image is a still image, even if the still image data and correction data are separately transmitted to the display device 800, there is a one-to-one correspondence between the still image data and the correction data, allowing the display device 800 to perform the correct optical correction processing and image stabilization processing. On the other hand, if the extracted image is a moving image, if the moving image data and correction data are separately transmitted to the display device 800, it becomes difficult to determine which correction data the transmitted correction data corresponds to for each frame of the moving image. In particular, if the clock rate of the overall control CPU 101 in the camera body 1 and the clock rate of the display device control unit 801 in the display device 800 are slightly different, synchronization between the overall control CPU 101 and the display device control unit 801 can be lost after several minutes of moving image capture. As a result, a problem occurs in which the display device control unit 801 corrects a frame to be processed with correction data that is different from the correction data corresponding to that frame.

[0302] Therefore, in this embodiment, when data of the moving image clipped from the camera body 1 is transmitted to the display device 800, the correction data is appropriately added to the data of the moving image. The method for doing this will be described below.

[0303] Figure 14 is a flowchart of the subroutine of the primary recording process of S600 in Figure 7A. This process will be described below with reference to Figure 15 as well. Figure 14 shows the process when the mode selected by the imaging mode switch 12 is the video mode. Note that when the selected mode is the still image mode, this process begins with the process of S601 and ends when the process of S606 is completed.

[0304] In S601a, the overall control CPU 101 reads out an image of one frame that has not yet been processed in S601 to S606 from the moving image developed in the recording range development process (FIG. 7E). The overall control CPU 101 also generates correction data, which is metadata for the read frame.

[0305] In S601, the overall control CPU 101 attaches information about the image cut-out position of the frame read in S600 to the correction data. The information attached here is the coordinates Xi, Yi of the video recording frame 127i acquired in S305. Note that the information attached here may also be vector information indicating the observation direction Vi.

[0306] In S602, the overall control CPU 101 acquires an optical correction value. The optical correction value is the optical distortion value set in S303. Alternatively, it may be a correction value according to the optical characteristics of the lens, such as a peripheral illumination correction value or a diffraction correction value.

[0307] In S603, the overall control CPU 101 attaches the optical correction value used in the distortion conversion in S602 to the correction data.

[0308] In S604, the overall control CPU 101 determines whether the camera is in the anti-shake mode. Specifically, if the preset anti-shake mode is "medium" or "strong," it is determined that the camera is in the anti-shake mode, and the process proceeds to S605. On the other hand, if the preset anti-shake mode is "off," it is determined that the camera is not in the anti-shake mode, and the process proceeds to S606. Note that the reason for skipping S605 if the anti-shake mode is "off" is that the amount of data calculated by the overall control CPU 101 and the amount of data transmitted wirelessly can be reduced by the amount of skipping, which in turn reduces the power consumption and heat generation of the camera body 1. Note that while the reduction of data used in the anti-shake process has been described above, data regarding the peripheral illumination correction value and the presence or absence of diffraction correction, which are included in the optical correction value acquired in S602, may also be reduced.

[0309] In this embodiment, the image stabilization mode is set in advance by the user's operation on the display device 800, but it may also be set as an initial setting on the camera body 1. Furthermore, in the case of a camera system in which the presence or absence of image stabilization processing is switched after transfer to the display device 800, S604 may be omitted, and the process may proceed directly from S603 to S605.

[0310] In S605, the overall control CPU 101 attaches the image stabilization mode acquired in S302 and the gyro data during video capture associated with the frame read out in S601a from the primary memory 813 to the correction data.

[0311] In S606, the overall control CPU 101 updates the video file 1000 (FIG. 15) with data obtained by encoding the image data of the frame read out in S600 and the correction data to which various data has been added in S601 to S605. Note that when the first frame of the moving image is read out in S601a, the video file 1000 is generated in S606.

[0312] In S607, the overall control CPU 101 determines whether reading of all frame images of the moving image developed in the recording range development process (FIG. 7E) has been completed, and if not, returns to S601a. ​​On the other hand, if completed, the process exits this subroutine. The generated video file 1000 is saved in the internal non-volatile memory 102. In addition to being saved in the primary memory 813 and the internal non-volatile memory 102 described above, it may also be saved in the large-capacity non-volatile memory 51. In addition, a transfer process (S700 in FIG. 7A) is executed to immediately transfer the generated video file 1000 to the display device 800. After being transferred to the display device 800, the video file 1000 may be saved in the primary memory 813.

[0313] In this embodiment, encoding refers to combining video data and correction data into a single file, but in this case, the video data may be compressed, or the combined video data and correction data may be compressed.

[0314] FIG. 15 is a diagram showing the data structure of the video file 1000.

[0315] A video file 1000 is made up of a header 1001 and a frame section 1002. The frame section 1002 is made up of a frame data set, which is a set of images of each frame that makes up the video and the corresponding frame metadata. In other words, the frame 1002 section contains as many frame data sets as there are frames in the video.

[0316] In this embodiment, the frame metadata is encoded information on the cut-out position (position information within the image), optical correction values, and correction data to which gyro data is attached as needed, but is not limited to this. For example, the amount of information in the frame metadata may be changed by adding other information to the frame metadata or deleting information from the frame metadata depending on the imaging mode selected by the imaging mode switch 12.

[0317] An offset value or a starting address to the frame data set of each frame is recorded in the header 1001. Alternatively, the header 1001 may store metadata such as the time and size corresponding to the video file 1000.

[0318] Thus, in the primary recording process (FIG. 14), a video file 1000 consisting of a set of each frame of the moving image developed in the recording range development process (FIG. 7E) and its metadata is transferred to the display device 800. Therefore, even if the clock rate of the overall control CPU 101 in the camera body 1 and the clock rate of the display device control unit 801 in the display device 800 are slightly different, the display device control unit 801 can reliably perform correction processing on the moving image developed in the camera body 1.

[0319] In this embodiment, the optical correction value is included in the frame metadata, but the optical correction value may be assigned to the entire video.

[0320] Fig. 16 is a flowchart of the subroutine of the transfer process to the display device 800 in S700 of Fig. 7A. Fig. 16 shows the process when the mode selected by the imaging mode switch 12 is the video mode. Note that when the selected mode is the still image mode, this process starts from the process of S702.

[0321] In S701, it is determined whether the video recording (S400) by the image capture unit 40 has finished or is still in progress. If video recording (video capture) is in progress, the frame-by-frame recording area development process (S500) and the update (S606) of the video file 1000 in the primary recording process (S600) are sequentially performed. Because wireless transfer places a heavy load on the power, performing it in parallel with video recording can require a larger battery capacity for the battery 94 or additional heat countermeasures. Furthermore, in terms of computing power, performing wireless transfer in parallel with video recording increases the computing load, necessitating the provision of a high-spec overall control CPU 101, which also increases costs. In light of these factors, the present embodiment waits for the video recording to finish (YES in S701), then proceeds to S702, where a connection with the display device 800 is established. However, if the camera system of this embodiment has sufficient power supplied from the battery 94 and no additional heat generation measures are required, the camera body 1 may be connected to the display device 800 in advance, such as when the camera body 1 is started up or before recording begins.

[0322] In S702, in order to transfer the video file 1000, which has a large amount of data, to the display device 800, a connection with the display device 800 is established via the high-speed wireless unit 72. The low-power wireless unit 71 is used to transfer a low-resolution video (or video) to the display device 800 to check the angle of view, and to send and receive various setting values ​​to and from the display device 800, but is not used to transfer the video file 1000 because it takes time for transmission.

[0323] In S703, the video file 1000 is transferred to the display device 800 via the high-speed wireless unit 72, and when the transfer is complete, the process proceeds to S704, where the connection with the display device 800 is closed, and the process exits this subroutine.

[0324] So far, we have explained the case where one video file containing images of all frames of one video is transferred, but in the case of a long video spanning several minutes, multiple video files separated by time may be used. With a video file having the data structure shown in Fig. 15, even if one video is transferred to display device 800 as multiple video files, it is possible for display device 800 to correct the video without timing misalignment with the correction data.

[0325] Fig. 17 is a flowchart of the optical correction process subroutine of S800 in Fig. 7A. This process will be described below with reference to Fig. 18. As described above, this process is executed by the display device control unit 801 of the display device 800.

[0326] In S801, first, the display device control unit 801 receives the video file 1000 from the camera body 1 that was transferred in the transfer process (S700) to the display device 800. Thereafter, the display device control unit 801 acquires the optical correction value extracted from the received video file 1000.

[0327] Next, in S802, the display device control unit 801 acquires a video (one frame image obtained by capturing a moving image) from the video file 1000.

[0328] In S803, the display device control unit 801 corrects the optical aberration of the image acquired in S802 using the optical correction value acquired in S801, and stores the corrected image in the primary memory 813. When performing optical correction, if cropping is performed from the image acquired in S802, the image is cropped and processed within a range of the image narrower than the development range (target field of view 125i) determined in S303 (cropped development area).

[0329] FIG. 18 is a diagram for explaining the process of performing distortion correction in S803 of FIG.

[0330] 18(a) is a diagram showing the position of a subject 1401 as seen by the user's naked eye when capturing an image, and FIG. 18(b) is a diagram showing the image of the subject 1401 captured on the solid-state imaging element 42. FIG.

[0331] Figure 18(c) is a diagram showing the development area 1402 in the image of Figure 18(b). Here, the development area 1402 is the cut-out development area explained above.

[0332] Fig. 18(d) is a diagram showing a cut-out developed image in which the image of the developed area 1402 has been cut out, and Fig. 18(e) is a diagram showing an image obtained by correcting the distortion of the cut-out developed image of Fig. 18(d). Because cut-out processing is performed when correcting the distortion of the cut-out developed image, the image shown in Fig. 18(e) has an even smaller angle of view than the cut-out developed image shown in Fig. 18(d).

[0333] Fig. 19 is a flowchart of the image stabilization process subroutine of S900 in Fig. 7A. This process will be described below with reference to Fig. 18(f). As mentioned above, this process is executed by the display device control unit 801 of the display device 800.

[0334] In S901, the display device control unit 801 acquires gyro data for the current frame and the previous frame and the blur amount Vn-1Det calculated for the previous frame in S902 (described later) from the frame metadata of the video file 1000. Then, an approximate blur amount VnPre is calculated from this information. Note that in this embodiment, the current frame is the frame currently being processed, and the previous frame is the frame immediately before the current frame.

[0335] In S902, the display device control unit 801 obtains a detailed amount of blur VnDet from the video file. The amount of blur is detected by calculating how much the feature points of the image in the current frame have moved from the previous frame.

[0336] Known methods can be used to extract feature points. For example, a luminance information image can be generated by extracting only the luminance information of a frame image, and then the resulting image, shifted by one to several pixels, can be subtracted from the original image, and pixels whose absolute values ​​are equal to or greater than a threshold can be extracted as feature points. Alternatively, the luminance information image can be subjected to a high-pass filter, and the resulting image can be subtracted from the original luminance information image, and the extracted edges can be extracted as feature points.

[0337] The difference is calculated multiple times while shifting the brightness information images of the current frame and the previous frame by one to several pixels, and the amount of movement is calculated by calculating the position where the difference in the pixels of the feature points becomes small.

[0338] Because multiple feature points are required, as described below, it is preferable to divide the images of the current and previous frames into multiple blocks and extract feature points for each block. The block division depends on the number of pixels and aspect ratio of the image, but generally, 12 blocks (4 x 3) or 54 blocks (9 x 6) are preferable. If the number of blocks is too small, it becomes difficult to accurately correct trapezoidal distortion caused by the tilt of the imaging unit 40 of the camera body 1 and rotational blur in the optical axis direction. However, if the number of blocks is too large, the size of each block becomes small, and feature points become closer together, resulting in errors. For these reasons, the optimal number of blocks is selected appropriately depending on the number of pixels, ease of finding feature points, the angle of view of the subject, and other factors.

[0339] Calculating the amount of movement requires multiple difference calculations, with the luminance information images of the current and previous frames shifted by one or several pixels, resulting in a large amount of calculation. However, since the actual amount of movement is calculated based on the coarse blur amount VnPre and the deviation from it (how many pixels it is shifted), it is possible to significantly reduce the amount of calculation by calculating the difference only in the vicinity of the coarse blur amount.

[0340] Next, in S903, the detailed shake amount VnDet acquired in S902 is used to perform image stabilization processing, and then the process exits from this subroutine.

[0341] Known methods of image stabilization include Euclidean transformation, which allows rotation and translation, affine transformation, which allows these, and projective transformation, which also allows trapezoidal correction.

[0342] Euclidean transformation can correct movement and rotation in the X-axis and Y-axis directions, but cannot correct blurring caused by camera shake in the forward / backward direction or in the pan / tilt direction that occurs in the imaging unit 40 of the camera body 1. Therefore, in this embodiment, stabilization processing is performed using affine transformation, which can also correct enlargement, skew, etc. The affine transformation, which moves the coordinates (x, y) of the reference feature point to the coordinates (x', y'), is expressed by the following equation 100.

[0343]

number

[0344] The affine coefficients of the 3x3 matrix in Equation 100 can be calculated if the offsets of at least three feature points can be detected. However, if the detected feature points are close to each other or lie on a straight line, the image stabilization process will be inaccurate for points farther away from the feature points or points away from that line. Therefore, it is preferable to select detected feature points that are far from each other and do not lie on a straight line. Therefore, if multiple feature points are detected, the close feature points are excluded and the remaining feature points are normalized using the least squares method.

[0345] Fig. 18(f) is a diagram showing an image obtained by performing the stabilization process of S903 on the distortion-corrected image shown in Fig. 18(e). Because clipping is performed during the stabilization process, the angle of view of the image shown in Fig. 18(f) is smaller than that of the image shown in Fig. 18(e).

[0346] By performing such image stabilization processing, it is possible to obtain high-quality images with corrected shake.

[0347] The series of operations executed by the camera body 1 and display device 800 included in the camera system of this embodiment have been described above.

[0348] When a user turns on power switch 11, selects video mode with imaging mode switch 12, and simply observes a scene straight ahead without turning his or her head up, down, left, or right, face direction detection unit 20 first detects observation direction vo (vector information [0°, 0°]) (FIG. 12A). Then, recording direction / angle of view determination unit 30 extracts an image of target field of view 125o (FIG. 11B) shown in FIG. 12A from the ultra-wide-angle image projected onto solid-state imaging element 42.

[0349] Thereafter, when the user starts observing, for example, a child (subject A131) in FIG. 11A without operating camera body 1, face direction detection unit 20 first detects observation direction vm (vector information [-42°, -40°]) (FIG. 11C). Then, recording direction / angle of view determination unit 30 cuts out an image with a target field of view of 125 m (FIG. 11C) from the ultra-wide-angle image captured by imaging unit 40.

[0350] In this way, optical correction processing and image stabilization processing are performed on display device 800 at S800 and S900 for images cropped into various shapes according to the observation direction. As a result, even if overall control CPU 101 of camera body 1 has low specifications, and an image with significant distortion, for example, an image with a target field of view of 125 m (FIG. 11C) is cropped, it is possible to produce an image with distortion and shaking corrected, centered on a child (subject A131), as shown in FIG. 11D. In other words, the user can obtain an image captured in their own observation direction without touching camera body 1, other than turning on power switch 11 and selecting a mode with imaging mode switch 12.

[0351] The pre-setting mode will now be described. As mentioned above, because camera body 1 is a small wearable device, it is not provided with operation switches or a setting screen for changing its detailed settings. For this reason, in this embodiment, the detailed settings of camera body 1 are changed on the setting screen (FIG. 13) of display device 800, which serves as an external device.

[0352] For example, consider a case where you want to capture the same video continuously at a 90° angle of view and a 45° angle of view. Because the angle of view is set to 90° in normal video mode, to capture such video, you must first capture the video in normal video mode, then stop the video capture, switch display device 800 to the setting screen of camera body 1, and switch the angle of view to 45°. However, during continuous capture, performing such an operation on display device 800 is cumbersome, and you may miss capturing the video you wanted to record.

[0353] On the other hand, if the pre-setting mode is set in advance to a mode for capturing video at a 45° angle of view, after capturing video at a 90° angle of view, the user can instantly switch to capturing video at a zoomed-in angle of 45° simply by sliding the capture mode switch 12 to "Pre." In other words, the user does not need to interrupt the current capture operation and perform the above-mentioned troublesome operation.

[0354] The settings made in the pre-setting mode may include not only changes to the angle of view, but also vibration damping levels specified as "strong," "medium," or "off," as well as voice recognition settings not described in this embodiment.

[0355] For example, in the previous imaging situation, if the user switches from video mode to pre-setting mode using imaging mode switch 12 while continuing to observe a child (subject A131), the field of view setting value V will change from 90° to 45°. In this case, recording direction / field of view determination unit 30 cuts out an image of a target field of view of 128 m, indicated by the dotted line frame in FIG. 11E, from the ultra-wide-angle image captured by imaging unit 40.

[0356] Even in the pre-setting mode, optical correction processing and image stabilization processing are performed in S800 and S900 on the display device 800. As a result, even if the overall control CPU 101 of the camera body 1 has low specifications, it is possible to obtain an image in which distortion and shaking have been corrected, zoomed in on the child (subject A131) as shown in FIG. 11F. While the example in which the angle of view setting value V is changed from 90° to 45° in video mode has been described, the same applies to still image mode. The same also applies when the angle of view setting value V for video is 90° and the angle of view setting value V for still images is 45°.

[0357] In this way, the user can obtain a zoomed-in image of the direction of his or her own observation simply by switching the mode using the imaging mode switch 12 on the camera body 1.

[0358] In order to reduce the processing load on the display device 800, the optical correction process (S800) and the image stabilization process (S900) that were previously performed by the display device 800 may be performed by the camera body 1.

[0359] Furthermore, in this embodiment, the case where the face direction detection unit 20 and the imaging unit 40 are integrally configured in the camera body 1 has been described. However, this is not limited to this, as long as the face direction detection unit 20 is worn somewhere on the user's body other than the head, and the imaging unit 40 is worn on the user's body. For example, the imaging and detection unit 10 of this embodiment can also be installed on the shoulder or abdomen. However, in the case of the shoulder, if the imaging unit 40 is installed on the right shoulder, it is possible that the subject on the left side will be blocked by the head, so a configuration that compensates by installing multiple imaging units, including on the left shoulder, is preferable. Furthermore, in the case of the abdomen, spatial parallax occurs between the imaging unit 40 and the head, so it is desirable to be able to perform a correction calculation of the observation direction to correct this parallax.

[0360] Next, a method for determining the amount of correction (calibration method) for correcting personal differences and adjustment differences of the user who wears camera body 1 will be described in detail with reference to FIGS.

[0361] Each user wearing camera body 1 has individual differences and adjustment differences, typified by factors such as physique, the tilt and angle of the area around the neck, the condition of clothing around the neck (such as the presence or absence of a collar), and the length of bands 82L and 82R. For this reason, the optical axis direction of imaging lens 16 of camera body 1 (or the center of the optical axis of the captured image) does not usually coincide with the line of sight when the user is facing forward (hereinafter referred to as the user's natural state). From the perspective of capturing an image of a scene observed by the user, it is desirable to align the center of target field of view 125, which is extracted from the captured image obtained by camera body 1, with the line of sight (center of field of view), which changes depending on the user's posture and movement.

[0362] Furthermore, there are individual differences not only in the center of the user's field of view when the user is in a natural state, but also in the center of the field of view when the user turns his / her head in various directions, including up, down, left, right, and diagonally, as well as in the movable range of the neck. Therefore, there are individual differences in the relationship between the face direction (observation direction) detected by the face direction detection unit 20 and the gaze direction. Therefore, in order to reduce the deviation between the center of the target field of view 125 and the user's gaze direction, calibration is required to determine the relationship (correction amount) between the face direction (observation direction) detected by the face direction detection unit 20 and the center of the target field of view 125 set based on the observation direction.

[0363] Typically, calibration can be performed as part of the preparatory operation process (S100) in FIG. 7A. It is generally assumed that calibration is performed when the camera body 1 is first started up, but it may also be performed when a certain amount of time has passed since calibration, or when the camera body 1 has shifted position relative to the user since the previous calibration. Calibration may also be performed when the face direction detection unit 20 can no longer detect the user's face. Furthermore, if it is detected that the user has removed the camera body 1, calibration may be performed when the user puts it back on. In this way, it is desirable to perform calibration appropriately at a timing determined to be necessary for the appropriate use of the camera body 1.

[0364] FIG. 20 is a diagram showing details of a calibrator 850 used for calibration. Here, it is assumed that the display device 800 shown in FIGS. 1D and 6 also serves as the calibrator 850. Therefore, in the following description, it is assumed that the calibrator 850 has the configuration shown in FIG. 6. However, the calibrator 850 may be a device separate from the display device 800. Alternatively, the calibration can be performed by executing an application that realizes the following operations on a general computer device with a camera function, such as a smartphone or tablet terminal.

[0365] The calibrator 850 has an A button 802, a display unit 803, an in-camera 805, a face sensor 806, and an angular velocity sensor 807. Furthermore, for example, a display device control unit 801 (hereinafter simply referred to as the control unit 801) executes a program to display a positioning index 851 and a calibration button 854 on the display unit 803. Note that the B button 804 is not essential and can be replaced by the calibration button 854 as will be described later, and is therefore not shown in FIG.

[0366] 20(a) shows a case where the positioning index 851 is a specific pattern displayed on the display unit 803, while Fig. 20(b) shows a case where the external appearance (casing) of the calibrator 850 is used as the positioning index 851. In the case of Fig. 20(b), the positioning index center 852, which will be described later, is calculated from information about the external shape of the calibrator 850.

[0367] Note that the positioning index 851 is not limited to the examples shown in Figures 20(a) and (b) and may be, for example, a separate entity from the calibrator 850. The positioning index 851 can be implemented in any manner as long as it has a shape that makes it easy to measure the size and is easy for the user to see. For example, the lens cap of the imaging lens 16 or the charging unit of the camera body 1 may be used as the positioning index 851. Because calibration does not depend on the shape of the positioning index 851, the following description will mainly assume the use of the positioning index 851 shown in Figure 20(a).

[0368] The positioning index 851 is a pattern of a specific shape that is displayed on the display unit 803 of the calibrator 850. The width L851a, height L851b, and positioning index center 852 of the positioning index are known or can be calculated. In order to prompt the user to observe the center of the positioning index 851 during the calibration process described below, it is desirable that the positioning index 851 have an appearance (shape and / or color) that makes the center easily visually identifiable. While FIG. 20(a) shows the positioning index 851 as a circular pattern with an emphasized center as an example, it is not limited to this shape. Other shapes that may be used include a square, triangle, or star, or an illustration of a character, for example.

[0369] The positioning index 851 is captured by the imaging unit 40 of the camera body 1. Based on the captured image, the control unit 801 calculates the distance between the imaging / detection unit 10 and the positioning index 851 (calibrator 850) and the coordinates of the positioning index 851 in the image. In this embodiment, these calculations are performed by the control unit 801, but if the calibrator 850 does not have a calculation function, the overall control CPU 101 of the camera body 1 can perform the calculations.

[0370] The angular velocity sensor 807 can measure the movement of the calibrator 850. Based on the measurement value of the angular velocity sensor 807, the control unit 801 calculates movement information indicating the position and orientation of the calibrator 850, which will be described later.

[0371] The calibration button 854 is a button that the user operates during the calibration process described below while gazing at the center of the positioning index 851. In Fig. 20(a), the calibration button 854 is a touch button (GUI) displayed on the touch panel display unit 803, but the A button 802 or the B button 804 may also function as the calibration button.

[0372] Next, the calibration process will be described in detail using the flowchart in Fig. 21. Fig. 21 is a flowchart of the calibration process according to this embodiment, which is executed by the camera body 1 and the calibrator 850. In Fig. 21, boxes are drawn to indicate the subject of the operation of each step.

[0373] 21, for the sake of convenience, the steps in which a user's operation is received by the camera body 1 or the calibrator 850 are described as being performed by the user. Furthermore, the steps in which the control unit 801 of the calibrator 850 executes in response to the received user's operation are described as being performed by the calibrator 850. Furthermore, the steps in which the overall control CPU 101 of the camera body 1 executes in response to the received user's operation are described as being performed by the camera body 1.

[0374] 21, steps S3104 and S3108 are performed by the camera body 1, steps S3101, S3105, and S3106 are performed by the user, and steps S3102, S3103, S3106a, S3107, S3107b, and S3110 are performed by the calibrator 850.

[0375] In S3101, if the power of the calibrator 850 is not ON, the user operates the A button 802 to turn ON the power of the calibrator 850. Similarly, if the power of the camera body 1 is not ON, the user switches the power switch 11 ON to turn ON the power of the camera body 1. The user then establishes a connection between the calibrator 850 and the camera body 1. Once this connection is established, the control unit 801 and the overall control CPU 101 each enter calibration mode.

[0376] Also, in S3101, the user puts on the camera body 1 and adjusts the length of the band portions 82L and 82R and the angle of the camera body 1, etc., to place the camera body 1 in a suitable position so that the imaging / detection unit 10 can take images.

[0377] In S3102, the control unit 801 displays a positioning index 851 on the display unit 803 by executing, for example, a calibration application.

[0378] Next, in S3103, the control unit 801 causes the display unit 803 to display an instruction display 855 as shown in FIG. 22A. The instruction display 855 includes a message instructing the user to a designated position over which the calibrator 850 should be held. In this embodiment, five positions are designated in this order: the front, the upper right, the lower right, the upper left, and the lower left. The front is a position where the center of the field of view (the line of sight) when the user is facing forward is the center of the positioning index 351. The upper right, the lower right, the upper left, and the lower left are relative positions with the front as the reference. However, the designated positions are not limited to these as long as calibration (calculation of the correction amount) between the observation direction and the line of sight is possible.

[0379] In S3104, the overall control CPU 101 activates the photographing unit 40 to put it into a state where it can take an image, and also activates the face direction detecting unit 20 to put it into a state where it can detect the face direction of the user.

[0380] In S3105, the user moves the calibrator 850 so that the positioning index 351 is visible at the specified position instructed in S3103. If the positioning index 351 is an object separate from the calibrator 850, the user places the positioning index 351 at the specified position.

[0381] Next, in S3106, the user turns his / her face toward the positioning index 851 while maintaining the position of the calibrator 850 at the specified position. Then, the user operates the calibration button 854 while gazing at the center of the positioning index 851 (aligning the center of the field of view with the center of the positioning index 851).

[0382] In S3106a, the control unit 801 determines whether or not the user looked at the positioning index center 852 at the center of the field of view. Details of the determination method will be described later. If the control unit 801 determines that the user looked at the positioning index center 852 at the center of the field of view, it notifies the user in S3107 with an instruction display 855 that calibration of the specified position will begin. Furthermore, the control unit 801 re-displays the calibration button 854. If the control unit 801 does not determine that the user looked at the positioning index center 852 at the center of the field of view, it again displays the instruction display 855 similar to S3103 and re-displays the calibration button 854.

[0383] When the user operates the calibration button 854 in S3107a, the control unit 801 sends a calibration instruction to the camera body 1 in S3107b.

[0384] In S3108, in response to a calibration instruction from the calibrator 850, the overall control CPU 101 acquires an ultra-wide-angle image in which the positioning index 851 is captured by the imaging unit 40, and detects the observation direction with the face direction detection unit 20. At this time, the face direction detection unit 20 measures the distance between the neck position 206 and the chin position 207. Thereafter, the overall control CPU 101 calculates position coordinate information of the positioning index center 852 in the acquired ultra-wide-angle image, and generates calibration data indicating the amount of correction for correcting a deviation between the calculated position coordinate information and the detected observation direction.

[0385] The details of the processes in S3103 to S3108 will be explained below with reference to FIGS. 22A to 22F.

[0386] 22A to 22F are diagrams for explaining calibration in the direction in front of the user. Calibration causes the center position of the field of view of the user in a natural state to coincide with the center position of target field of view 125 in the image captured by imaging unit 40 of camera body 1.

[0387] FIG. 22A is a diagram showing a screen displayed on display unit 803 of calibrator 850 in S3103 of FIG. 21 during calibration in the front direction of the user.

[0388] As shown in FIG. 22A, a display unit 803 of a calibrator 850 displays a positioning index 851 and an instruction display 855 that shows the user where to place the positioning index 851.

[0389] The instruction display 855 is a character string that instructs the user to place the positioning index 851 at the center of the field of view when the user faces forward. Note that the instruction displayed as the instruction display 855 is not limited to text, and may be in the form of other instructions such as illustrations, photographs, or videos.

[0390] Alternatively, an instruction display 855 may be displayed, followed by the positioning indicator 851, as in a so-called general tutorial.

[0391] FIG. 22B is a diagram showing a state in which a user holds the calibrator in front of them in accordance with the instructions shown on the instruction display in FIG. 22A.

[0392] The user holds the calibrator 850 in front of them according to the instructions shown in the instruction display 855 in Fig. 22A (S3105). Then, the user positions the calibrator 850 so that the positioning index 851 is at the center of the field of view when facing the front of the face, and presses the calibration button 854 (Fig. 22A) (S3106). In response to pressing the calibration button 854, a determination is made in S3106a. The specific procedure of this determination method will be described later. If the determination in S3106a is YES, the control unit 801 changes the instruction display 855 shown in Fig. 22A to a notification saying "Starting calibration in the front direction" in S3107, and redisplays the calibration button 854.

[0393] Thereafter, the user confirms that the instruction display 855 shown in FIG. 22A has changed to a notice saying "Starting calibration in the front direction," and then presses the calibration button 854 (S3107a).

[0394] In response to pressing of this calibration button 854, a calibration instruction is sent to the camera body 1 in S3107b, and the image capturing section 40 acquires the captured image in S3108.

[0395] FIG. 22C is a schematic diagram showing the entire ultra-wide-angle image captured by imaging lens 16 in the state of FIG. 22B, and FIG. 22D is a schematic diagram showing an image in which the aberration of the ultra-wide-angle image shown in FIG. 22C has been corrected.

[0396] On the other hand, in response to the user pressing the calibration button 854 in the state shown in FIG. 22B, the face direction detection unit 20 acquires the observation direction in S3108.

[0397] FIG. 22E is a schematic diagram showing a face direction image recorded by the face direction detection unit 20 in S3108 of FIG. 21 during calibration for the front direction of the user.

[0398] As described above with reference to FIGS. 8G to 8K, the face direction detection unit 20 calculates the angles in the left-right and up-down directions of the face using the distances and angles between the chin positions 207, 207r, 207u, etc. and the neck position 206. However, like the image center, the distances and angles between the chin positions 207, 207r, 207u, etc. and the neck position 206 are also not constant due to individual differences and adjustment differences, such as those expressed by the user's physique. Therefore, the distance between the chin position 207 and the neck position 206 at the time the calibration button 854 is operated is saved as a value when the user sets the front as the center of the user's field of view. This makes it possible to accurately calculate the user's field of view center (gaze direction) regardless of individual differences and adjustment differences. Note that the distances are not limited to the chin position 207 and the neck position 206, but any two positions that can be detected as feature points of the head from the captured image can be used.

[0399] 21, in S3109, the overall control CPU 101 determines whether preparation for calibration in the front direction is complete or not. That is, it determines whether information necessary for calculating the chin position 207, the neck position 206, and the positioning index center 852 has been acquired or not.

[0400] At this time, if the necessary information has not been acquired, it is determined that preparation for calibration in the front direction is not complete (NO in S3109), and the operations from S3102 are repeated so that the missing information among the necessary information can be acquired again. Note that, if the necessary information has not been acquired, it is not necessary to perform all the operations from S3102 onwards, and only the operations necessary to re-acquire the missing information can be performed again.

[0401] The determination in S3106a described above can be made using the face sensor 806 or the in-camera 805 of the calibrator 850. Specific steps of this determination method will be described below, taking as an example a case where the in-camera 805 is used to calibrate the front direction of the user. Note that a description of the case where the face sensor 806 is used is omitted, as the basic concept is the same, although there is a difference between two-dimensional and three-dimensional information. However, when the face sensor 806 is used in the determination in S3106a, the face direction detection unit 20 of the camera body 1 does not perform face detection by projecting infrared rays 23 at the user while the face sensor 806 is projecting infrared rays 823 at the user. This is to prevent the infrared rays 23 and 823 from interfering with each other.

[0402] First, when the user operates calibration button 854 in Fig. 22A in S3106, control unit 801 captures an image with in-camera 805 and acquires in-camera video 858 (Fig. 22F) showing the user. Furthermore, control unit 801 detects, from acquired in-camera video 858, the user's front neck 201, chin 203, and face 204 including the nose, as well as position information of imaging / detection unit 10 (imaging unit 40).

[0403] Using each piece of position information detected in this in-camera image 858, the control unit 801 determines in S3106a whether the user is looking at the positioning index center 852 of the positioning index 851 at the center of the field of view.

[0404] If it is determined that the user is looking in a different direction, the control unit 801 displays information that correct information cannot be acquired on the instruction display 855. This allows the control unit 801 to instruct the user to perform calibration again.

[0405] Note that there are cases where the control unit 801 can determine, using the in-camera image 858, that correct calibration is not possible, such as when the image capturing / detection unit 10 is tilted to a certain degree or more, or when the face direction detection window 13 is blocked or dirty. In such cases, the control unit 801 may also display information on the instruction display 855 indicating that correct information cannot be acquired.

[0406] Returning to FIG. 21, if the overall control CPU 101 determines in S3109 that the necessary information has been acquired and that preparation for calibration in the front direction has been completed using the method described above, the process proceeds to S3110.

[0407] In S3110, the control unit 801 (correction means) calculates the correction amount for correcting the cutout center position so as to absorb individual differences and adjustment differences, and corrects (offsets) the cutout center position based on that information.

[0408] The specific calculation in S3110 is as follows.

[0409] If the user is in an ideal state according to the design values ​​and is wearing camera body 1 ideally, the position of center 856 of the ultra-wide-angle image acquired in S3108 shown in Fig. 22C should approximately match the position of positioning index center 852 appearing in that ultra-wide-angle image. However, in reality, due to individual differences and adjustment differences such as those expressed by the user's physique, the positions of center 856 in the ultra-wide-angle image and positioning index center 852 usually do not match.

[0410] For the user, it is desirable that the cropping center position is not the center 856 of the ultra-wide-angle image displayed by the camera body 1, but the center of the field of view in the user's posture and movement, that is, the position of the positioning index center 852 in the ultra-wide-angle image.

[0411] Therefore, the amount of deviation between the positioning index center 852 and the center 856 in the super wide-angle image is measured as the correction amount. Then, the cropping center position is offset from the center 856 of the super wide-angle image to a position based on the positioning index center 852 according to the correction amount. The face direction detected by the face direction detection unit 20 at that time is also offset in the same way.

[0412] A specific offset method will be described with reference to Figures 22C and 22D. As shown in Figure 22C, the amount of deviation of positioning index center 852 from center 856 of the ultra-wide-angle image is measured, and this is divided into horizontal deviation amount 857a and vertical deviation amount 857b. An appropriate conversion process according to the projection method of the full angle of view can be applied to obtain correction amounts for each direction.

[0413] Alternatively, as shown in FIG. 22D, after applying a conversion process to project an ultra-wide-angle image into an appropriate angle of view range, the correction amounts may be the horizontal shift amount 857c and the vertical shift amount 857d between the center 856a of the converted image and the positioning index center 852.

[0414] Whether to use the offset method shown in FIG. 22C or FIG. 22D can be determined arbitrarily in consideration of the processing load and purpose of the camera system.

[0415] By performing the above-described calibration of the front direction, it becomes possible to appropriately associate the face direction of each user when wearing the device with the center of the field of view for that face direction in the ultra-wide-angle image, and the face direction detected by the face direction detection unit 20, regardless of individual differences, adjustment differences, etc.

[0416] Up to this point, we have explained the calibration of the front direction out of the five directions: front, upper right, lower right, upper left, and lower left. However, similar calibration must also be performed for the four directions: upper right, lower right, upper left, and lower left.

[0417] Therefore, in FIG. 21, when the processing of S3110 is completed, the process proceeds to S3111.

[0418] In S3111, if it is determined that there is a direction among the five directions (front, upper right, lower right, upper left, and lower left) for which calibration has not yet been performed, the direction for which calibration is to be performed is changed to that direction, and the process returns to S3103. As a result, calibration is repeated in the same manner for the remaining directions other than the front direction for which calibration has already been completed.

[0419] Although not shown in FIG. 21, if it is determined in S3111 that there is no direction for which calibration has not been performed, this process ends.

[0420] 23A to 23E are diagrams for explaining calibration in the direction above the user's right hand (the direction above and to the right in an ultra-wide-angle image). Figures 23A to 23E correspond to Figures 22A to 22E, respectively, and the basic operations are the same, so common explanations will be omitted.

[0421] As shown in FIG. 23A, the instruction display 855 displays a written instruction to position the positioning index 851 at the center of the field of view when the user turns his / her face to the upper right.

[0422] FIG. 23B is a diagram showing a state in which the user holds the calibrator 850 upright in accordance with the instruction shown on the instruction display 855 in FIG. 23A.

[0423] FIG. 23C is a schematic diagram showing the entire super-wide-angle image captured by the imaging lens 16 in the state of FIG. 23B.

[0424] 23C, a specific offset method involves first measuring the amount of deviation between the center 856 and the positioning index center 852 in the ultra-wide-angle image. The measured deviation is then divided into a radial deviation 857e and an angular deviation 857f, and conversion processing for projecting the ultra-wide-angle image is applied to each deviation to obtain a correction amount.

[0425] Alternatively, as shown in FIG. 23D, after applying a conversion process to project an ultra-wide-angle image into a specific angle of view range, the correction amounts may be the radial deviation 857g and angular deviation 857h between the center 856a of the converted image and the positioning index center 852.

[0426] 22A to 22E, the amount of deviation was divided into the vertical and horizontal directions. In contrast, the amount of deviation was divided into the diameter and angular directions in the amount of deviation determined using FIGS. 23A to 23D, but this difference between these methods is merely for the sake of convenience, and either method may be used.

[0427] 23E, the face direction detection unit 20 can acquire the neck position 206 and the chin position 207ru required to calculate the face direction when the user looks to the upper right. Therefore, regardless of the individual differences and adjustment differences of the user, it is possible to correctly measure the face direction when the user looks in the direction of the positioning index center 852 (to the upper right in this case).

[0428] 21, calibration is performed not only in the front direction but also in the upper right, lower right, upper left, and lower left directions, and the correction amount for each direction is saved as calibration data for the user. This enables face direction detection unit 20 to correctly measure the center of the user's field of view (gaze direction) when the user turns their head in any direction, up, down, left, or right, and enables camera body 1 to appropriately capture the user's observation range regardless of individual differences or adjustment differences.

[0429] In the above description, for simplicity, a method of repeatedly performing calibration in five directions, i.e., the front, upper right, lower right, upper left, and lower left, has been explicitly described.

[0430] However, calibration is not limited to this method. For example, a method may be employed in which the user continuously moves the calibrator 850 along a Z-shaped, spiral, polygonal, or other trajectory in accordance with the instruction display 855 while simultaneously keeping the positioning index 851 displayed on the calibrator 850 at the center of the field of view. In this method, the control unit 801 transmits calibration instructions to the camera body 1 multiple times while the calibrator 850 is moving in this manner. Each time a calibration instruction is received, the overall control CPU 101 acquires the face direction detected by the face direction detection unit 20 and position coordinate information of the positioning index center 852 in the ultra-wide-angle image captured by the imaging unit 40, and stores these as history information. Thereafter, the overall control CPU 101 combines information extracted from the acquired history information to calculate the relationship between the center position of the image cropping and the user's face direction. Furthermore, using information acquired by the in-camera 805 or face sensor 806 on the calibrator 850 side while the calibrator 850 is moving using this method, the information extracted from the history information may be limited to information in a state where the user is looking at the positioning index 851. This prevents information in a state where the user is looking away from being extracted from the history information, thereby making it possible to improve the accuracy of relationship calculation.

[0431] Furthermore, the control unit 801 may also transmit the measurement values ​​from the angular velocity sensor 807 to the camera body 1 when issuing a calibration instruction. In this case, the overall control CPU 101 acquires movement information indicating the method of movement of the calibrator 850 by the user and the position and attitude of the calibrator 850 from the transmitted measurement values ​​from the angular velocity sensor 807, and stores this as history information. This makes it possible to perform calibration simply and accurately from the movement information based on the measurement values ​​from the angular velocity sensor 807, the observation direction detected by the face direction detection unit 20, and the position coordinate information of the positioning index center 852 in the ultra-wide-angle image captured by the imaging unit 40.

[0432] In this case, however, it is necessary that the movement information based on the measurement value by the angular velocity sensor 807 matches the movement information based on the position coordinate information of the positioning index 851. Therefore, when using the measurement value by the angular velocity sensor 807, it is necessary to synchronize the communication between the camera body 1 and the calibrator 850.

[0433] The above describes the details of the calibration process that calculates the amount of correction to correct the deviation between the detected observation direction and the actual center of gaze (gaze direction) due to individual differences and adjustment errors between users, and stores the correction data in association with the user. By determining the center position of the target field of view 125 in an ultra-wide-angle image using the calibration data associated with the user, it becomes possible to obtain an image with little difference between the center of gaze and the center position of the target field of view 125, even if the user changes. Note that the above-described calibration process is one example, and various modifications and changes are possible as long as similar calibration data can be obtained.

[0434] ●(Use of saved calibration data) We have explained a method for generating calibration data for a user wearing camera body 1. Calibration data is data specific to each user. Therefore, if there is more than one user of camera body 1, it is necessary to use calibration data that is appropriate for the actual user.

[0435] As with the implementation of the calibration process, the following description will be given assuming that the display device 800 has the functionality of the calibrator 850. Also, it is assumed that the power-saving wireless unit 71 of the camera body 1 and the power-saving wireless unit 871 of the calibrator 850 communicate in accordance with Bluetooth Low Energy (BLE).

[0436] The low-power radio unit 871 has a BLE BD (Bluetooth Device) address. The high-speed radio unit 872 has a Wi-Fi MAC (Media Access Control) address. If the calibrator 850 is a wireless communication terminal that can connect to a public communication line, such as a smartphone, it has an International Mobile Equipment Identity (IMEI).

[0437] The BD address, MAC address, IMEI, or a character string obtained by hashing any of these can be used as the unique ID or identification information of the calibrator 850. Note that other information may be used as the unique ID as long as it is information that can identify an individual calibrator 850.

[0438] In this embodiment, the calibration data generated in the calibration process is stored in the built-in non-volatile memory 102 of the camera body 1. The camera body 1 can store multiple calibration data, and each calibration data is associated with the unique ID of the calibrator 850 used when it was generated and the name of the user.

[0439] The calibration process is performed using a calibrator 850 owned by the user who wears the camera body 1. Therefore, the unique ID of the calibrator 850 is information related to the user of the calibrator 850. Therefore, it can be said that each piece of calibration data is associated with information about the user who was wearing the camera body 1 at the time of generation. In this embodiment, one calibrator is not shared by multiple people, and the unique ID of the calibrator 850 and the user's identification information are considered to be synonymous.

[0440] Table 1 shows an example in which the calibration data of two users, specifically the father (calibrator's unique ID = 0x0001) and the mother (calibrator's unique ID = 0x0002), is registered in the camera body 1 along with the date and time the calibration was performed. Note that in Table 1, the calibrator's unique ID is illustratively a four-digit hexadecimal number, but there are no restrictions on the number of digits or format of the unique ID. Note that the sign of the correction amount is a sign in a Cartesian coordinate system with the center of the image as the origin. In other words, the correction amount in the right and upward directions is positive, and the correction amount in the left and downward directions is negative. [Table 1]

[0441] The process for identifying the calibration data that should be used by the camera body 1 will be described using the flowchart shown in Figure 24. This process is executed by the overall control CPU 101.

[0442] First, the overall process will be described with reference to Fig. 24A. The process starts when the overall control CPU 101 detects that the power switch 11 of the camera body 1 has been operated to turn on the power to the camera body 1.

[0443] In S40001, the overall control CPU 101 receives the unique ID from the calibrator 850 that has requested connection. Details of this processing will be described later.

[0444] In S40002, the overall control CPU 101 determines whether or not it has received unique IDs from all calibrators 850 that detected connection requests in S40001. If it is determined that it has received unique IDs from all calibrators 850 that detected connection requests, the overall control CPU 101 executes S40003. Furthermore, if it is not determined that it has received unique IDs from all calibrators 850 that detected connection requests, the overall control CPU 101 executes S40001 for the calibrators 850 that have not received unique IDs.

[0445] In S40003, the overall control CPU 101 determines the priority of the calibration unique IDs to be used in S40004. That is, if there are multiple unique IDs received in S40001 and S40002, the overall control CPU 101 determines the priority of the unique IDs. Details of the processing will be described later. Note that if there is only one unique ID received in S40001, S40003 may be skipped.

[0446] In S40004, the overall control CPU 101 identifies the calibration data associated with the unique ID with the highest priority determined in S40003. Then, the overall control CPU 101 (acquisition means) reads the identified calibration data from the built-in nonvolatile memory 102 (storage means) to, for example, the primary memory 103.

[0447] In S40005, the overall control CPU 101 determines whether or not the calibration data identified in S40004 exists in the built-in non-volatile memory 102 (whether or not it was able to be read from the built-in non-volatile memory 102). If it is determined that the identified calibration data exists in the built-in non-volatile memory 102, the overall control CPU 101 executes S40007, and if not, executes S40006.

[0448] In S40006, the overall control CPU 101 requests the calibrator 850 having the unique ID with the highest priority determined in S40003 to transmit calibration data. Then, the overall control CPU 101 stores the calibration data received from the calibrator 850 in the primary memory 103. Thereafter, the overall control CPU 101 executes S40007.

[0449] In S40007, the overall control CPU 101 executes normal camera operation. Normal camera operation involves capturing an image using the camera body 1 and extracting and recording the target field of view 125 from the captured image. The overall control CPU 101 determines the center position of the target field of view 125 using the calibration data acquired in S40004 or S40006.

[0450] The details of the unique ID reception process executed in S40001 will be described using the flowchart shown in FIG. 24B.

[0451] In S41001, the overall control CPU 101 broadcasts an advertising packet through the power-saving wireless unit 71. The purpose of this process is to search for a calibrator 850 that exists within the communication range. The overall control CPU 101 may perform other operations depending on the protocol used for communication with the calibrator 850. Here, the power-saving wireless unit 71 performs communication in accordance with BLE. Furthermore, the power-saving wireless unit operates as a peripheral in BLE, and the power-saving wireless unit 871 of the calibrator 850 operates as a central in BLE.

[0452] In S41002, the overall control CPU 101 determines whether or not a connection request has been received from the calibrator 850, and if it is determined that a connection request has been received, executes S41003, and if not, executes S40001.

[0453] While in standby mode, the power-saving wireless unit 871 of the calibrator 850 periodically scans for advertising packets. When the calibrator 850 (control unit 801) detects an advertising packet from the camera body 1, it recognizes that the power of the camera body 1 has been turned on. Then, when the control unit 801 receives the advertising packet, it transmits a connection request to the camera body 1 via the power-saving wireless unit 871.

[0454] The transmission cycle of the advertising packets by the power-saving wireless unit 71 can be set appropriately within a range of, for example, 50 milliseconds to 1 second, but is not limited to this range. Also, although not shown in Fig. 24B, if a connection request is not received even after a certain time has elapsed since the first advertising packet was transmitted, the overall control CPU 101 may execute a timeout (connection failure) process.

[0455] Note that even if a calibrator 850 is present within the communication range of the power-saving wireless unit 71, if it is not paired with the camera body 1, it will not transmit a connection request in response to receiving an advertising packet. Furthermore, a calibrator 850 that does not have a predetermined application (for example, a calibration application or a display application) installed will also not transmit a connection request in response to receiving an advertising packet. The calibrator 850 used when generating calibration data is assumed to satisfy the conditions necessary for transmitting a connection request in response to receiving an advertising packet.

[0456] In S41003, the overall control CPU 101 switches the communication mode of the power-saving wireless unit 71 from broadcast mode to GATT (Generic Attribute Profile) communication mode, which performs one-to-one communication. The calibrator 850 (control unit 801) notifies the camera body 1 of the services and information (characteristics) that can be provided.

[0457] In S41004, the overall control CPU 101 requests the calibrator 850 to read the unique ID. Thereafter, the overall control CPU 101 receives the unique ID transmitted in response to the request from the calibrator 850. The overall control CPU 101 also acquires, from the calibrator 850, the received signal strength indicator (RSSI) and information about applications running in the foreground. This concludes the details of the unique ID reception process.

[0458] Here, the description has been given of the operation of receiving a unique ID from one calibrator 850. When connection requests are received from multiple calibrators 850, the overall control CPU 101 receives a unique ID from each calibrator that has sent a connection request.

[0459] Next, the unique ID priority order determination process executed in S40003 will be described in detail with reference to the flowchart shown in FIG. 24C.

[0460] In S42001, the overall control CPU 101 groups or classifies the multiple calibrators 850 corresponding to the unique IDs received in S40001 based on the execution state of the applications. Specifically, the overall control CPU 101 groups the calibrators into the following three groups: First group: Calibrators running an application (for example, a display application or a calibration application) related to cooperation with the camera body 1 Second group: Calibrator with no running applications or sleeping Third group: Calibrators running only applications not related to the linked operation with the camera body 1 (for example, applications other than the display application and the calibration application)

[0461] In S42002, the overall control CPU 101 sorts the calibrators belonging to the first group in descending order of RSSI strength and assigns higher priority to the calibrator with the strongest RSSI. If there are no calibrators belonging to the first group, the overall control CPU 101 skips S42002.

[0462] In S42003, the overall control CPU 101 sorts the calibrators belonging to the second group in descending order of RSSI strength, and sequentially assigns priorities lower than the already assigned priorities to the calibrator with the strongest RSSI. If there are no calibrators belonging to the first group, the calibrator with the strongest RSSI belonging to the second group will have the highest priority. If there are no calibrators belonging to the second group, the overall control CPU 101 skips S42003.

[0463] In S42004, the overall control CPU 101 sorts the calibrators belonging to the third group in descending order of RSSI strength, and assigns priorities lower than the already assigned priorities. If there are no calibrators belonging to the first and second groups, the calibrator with the strongest RSSI belonging to the third group will have the highest priority. If there are no calibrators belonging to the third group, the overall control CPU 101 skips S42004.

[0464] The overall control CPU 101 associates the priorities assigned in S42002 to S42004 with the unique IDs and stores them in, for example, the primary memory 103. The above is the details of the unique ID priority order determination process.

[0465] In this embodiment, the priority of the unique IDs is determined based on two perspectives. The first perspective is that the closer the calibrator 850 is to the camera body 1, the more likely it is that the calibrator 850 belongs to the user who is wearing the camera body 1. Therefore, the closer the distance to the camera body 1, the higher the priority assigned.

[0466] The second viewpoint is that a calibrator 850 that is running an application related to cooperative operation with the camera body 1 in the foreground is likely to be a calibrator 850 owned by the user who is wearing the camera body 1. A calibrator that is running a display application for checking the angle of view of the camera body 1 or for remote control, or a calibration application for the camera body 1 in the foreground, is given a higher priority than a calibrator that is not running such an application.

[0467] Table 2(a) shows an example of data received by the camera body 1 (overall control CPU 101) from two calibrators. [Table 2(a)]

[0468] In this case, neither calibrator is running an application related to cooperative operation with the camera body 1 in the foreground. Therefore, in S42001, the overall control CPU 101 groups both calibrators into the second group. Then, the overall control CPU 101 skips S42002, and in S42003 assigns priorities in descending order of RSSI. Therefore, the calibrator with unique ID 0x0001 has the highest priority, and the calibrator with unique ID 0x0002 has the second highest priority. Then, the overall control CPU 101 skips S42003, and the priorities are determined.

[0469] Table 2(b) shows another example of data received by the camera body 1 (overall control CPU 101) from two calibrators. [Table 2(b)]

[0470] In this case, in S42001, the overall control CPU 101 classifies the calibrator with unique ID 0x0001 into the third group and the calibrator with unique ID 0x0002 into the first group. As a result, the calibrator with unique ID 0x0002 has the highest priority, and the calibrator with unique ID 0x0001 has the second highest priority. The calibrator with unique ID 0x0001 has a stronger RSSI and is considered to be closer to the camera body 1 than the calibrator with unique ID 0x0001. However, based on the type of application being executed, the calibrator with unique ID 0x0002 is considered to be more likely to be the calibrator used by the user wearing the camera body 1, so it is given a higher priority.

[0471] Here, the priority is determined based on the RSSI and the type of foreground application, but this is just an example, and the priority may be determined based on other criteria.

[0472] For example, with Bluetooth (registered trademark) 5.1 or later, it is possible to detect the direction of the other device from the reception angle AoA (Angle of Arrival) and emission angle AoD (Angle of Departure). Therefore, based on the reception angle, emission angle, and RSSI, overall control CPU 101 can give a higher priority to calibrators that are likely to be held in the hand of the user wearing camera body 1.

[0473] For example, it is considered that a calibrator 850 that is significantly farther from camera body 1 than the arm's length of an average person and that is in the shooting direction of camera body 1 is unlikely to be held in the hand of a user wearing camera body 1. Conversely, it is considered that a calibrator 850 that is about the arm's length from camera body 1 and that is not in the shooting direction of camera body 1 is likely to be held in the hand of a user wearing camera body 1.

[0474] Next, details of the calibration request transmission process in S40006 will be described. S40006 is executed when the calibration data identified in S40004 does not exist in the camera body 1. In S40006, the overall control CPU 101 requests the calibrator 850 corresponding to the unique ID with the highest priority determined in S40003 to transmit the calibration data.

[0475] Upon receiving the request, the calibrator 850 (control unit 801) transmits the calibration data stored in the built-in non-volatile memory 812 to the camera body 1. The calibration data is a correction amount, and since the data volume is small, it may be transmitted via BLE GATT communication. On the other hand, if the calibration data includes, in addition to the correction amount, history information including the above-mentioned movement information, image data acquired during calibration, etc., the data volume will be large. If the calibration data is large, other communication methods may be used, such as using the high-speed wireless unit 72 or communication called Bluetooth Classic. The method used to communicate the calibration data can be determined taking into account the communication time lag, data transfer volume, communication speed, etc.

[0476] If the calibrator 850 that received the request does not hold calibration data, the control unit 801 prompts the user to perform calibration. The calibration method is as described above. If calibration is performed, the calibration data is saved in the camera body 1, and the overall control CPU 101 can use that calibration data. On the other hand, if calibration is not performed, the control unit 801 sends information indicating that there is no calibration data (error) or calibration data with a correction amount of 0 to the camera body 1. Then, in the case of an error, the overall control CPU 101 uses, for example, the calibration data used last time.

[0477] Note that even if it is determined in S40005 that calibration data exists in the camera body 1, S40006 may be executed if the calibrator holds newer calibration data. Specifically, if the calibration date and time included in the data received from the calibrator is later than the calibration date and time at which the calibration data stored in the camera body 1 was generated, the overall control CPU 101 may execute S40006. This allows newer calibration data to be used. Since newer calibration data is generally considered to have higher accuracy, an improvement in the positional accuracy of the target field of view is expected.

[0478] In this embodiment, an example has been described in which the calibration data used by the camera body 1 is identified when the power switch 11 of the camera body 1 is turned on. However, the calibration data may also be identified when other conditions are met, for example, when an instruction to start recording is detected.

[0479] According to this embodiment, it is possible to use calibration data appropriate for each user, without the need to perform calibration each time a different user uses the camera body 1. This not only improves usability, but also makes it possible to record images with an appropriate shooting range for each user.

[0480] ●(Second embodiment) Next, a second embodiment of the present invention will be described. In the first embodiment, the user who wears the camera body 1 owns the calibrator 850. However, there are cases where multiple people share one calibrator 850. For example, this may be the case when a child performs calibration using a calibrator owned by their parent. This embodiment relates to the handling of calibration data when one calibrator is shared by multiple people.

[0481] Since this embodiment can be implemented using the equipment described in the first embodiment, repeated explanation of the equipment configuration and other details described in the first embodiment will be omitted. Also, similar to the first embodiment, the display device 800 that executes the calibration application functions as the calibrator 850.

[0482] Table 3 shows an example of calibration data stored in the built-in non-volatile memory 102 of the camera body 1. In the example shown in Table 3, calibration data for different users (older brother and younger sister) exists for the same unique ID (0x0003). Note that in this embodiment, the calibration data is also associated with the distance between the neck position 206 and the chin position 207 measured during calibration. [Table 3]

[0483] Unlike the first embodiment, when one calibrator can be shared by multiple people, the calibrator 850 used for calibration is not necessarily located near the user wearing the camera body 1.

[0484] For this reason, in this embodiment, the overall control CPU 101 checks the unique ID of the calibrator associated with the calibration data stored in the built-in nonvolatile memory 102 of the camera body 1. If there is a plurality of calibration data associated with the same unique ID, the priority is determined without using the RSSI strength.

[0485] Details of the calibration data specification process in this embodiment will be described below with reference to the flowchart shown in Fig. 25. This process is executed in place of S40003 and S40004 in Fig. 24A described in the first embodiment. Other processes described with reference to Figs. 24A to 24C are the same as those in the first embodiment.

[0486] In S50001, the overall control CPU 101 executes the face direction detection process as described in S200 of FIG. 7A in the first embodiment.

[0487] In S50002, the overall control CPU 101 measures the distance between the neck position 206 and the chin position 207. According to the inventor's study, in the case of a camera shaped as shown in FIG. 1A, it has been found that immediately after attaching the camera to the neck, the user often turns to face forward psychologically in order to check the wearing status. Therefore, the distance measured in S50002 can be the distance when the user faces forward. Note that the distance is not limited to facing forward, as long as it is possible to measure the distance between the neck position 206 and the chin position 207 when facing a specific direction.

[0488] In S50003, the overall control CPU 101 classifies the calibrators 850 into first to third groups, similar to S42001 in the first embodiment.

[0489] In S50004, the overall control CPU 101 sorts the calibration data associated with the unique IDs of the calibrators belonging to the first group in order of the distance between the associated neck position and chin position closest to the distance measured in S50002.The overall control CPU 101 then assigns higher priority to the calibration data with closer distances.If there are no calibrators belonging to the first group, the overall control CPU 101 skips S42002.

[0490] In S50005, the overall control CPU 101 sorts the calibration data associated with the unique IDs of the calibrators belonging to the second group in order of the distance between the associated neck position and chin position closest to the distance measured in S50002.The overall control CPU 101 then assigns a lower priority than the already assigned priority to the calibration data with the closest distance.If there are no calibrators belonging to the second group, the overall control CPU 101 skips S50005.

[0491] In S50006, the overall control CPU 101 sorts the calibration data associated with the unique IDs of the calibrators belonging to the third group in order of the distance between the associated neck position and chin position closest to the distance measured in S50002.The overall control CPU 101 then assigns a lower priority than the already assigned priority to the calibration data with the closest distance.If there are no calibrators belonging to the third group, the overall control CPU 101 skips S50006.These are the details of the calibration data identification process in this embodiment.

[0492] If there is a calibrator that belongs to the first group, a screen for selecting the calibration data to be used may be displayed for the calibrator with the strongest RSSI, for example. In the example shown in Table 3, the overall control CPU 101 causes the calibrator 850 to display a confirmation dialog for selecting whether to use the older brother's or younger sister's calibration data. Then, the control unit 801 transmits information about the calibration data selected by the user to the camera body 1. The overall control CPU 101 determines the calibration data to be used based on the received information.

[0493] Furthermore, as explained in Table 2(b) of the first embodiment, when one unique ID is associated with one calibration data, the ranking based on the running application takes precedence over the distance between the camera body 1 and the calibrator 850. However, when there are multiple calibration data associated with the same unique ID, as in this embodiment, grouping based on the running application may not be performed. Alternatively, the second and third groups may not be distinguished.

[0494] According to this embodiment, even if multiple people share one calibrator, i.e., even if there are multiple calibration data associated with the same unique ID, it is possible to identify the calibration data suitable for the user wearing the camera body 1.

[0495] ●(Third embodiment) Next, a third embodiment of the present invention will be described. In the first and second embodiments, the handling of calibration data for correcting the deviation between the center of the user's field of view and the center position of the crop range of the captured image when the camera body 1 is a wearable device has been described. However, the present invention can also be applied to other calibration data.

[0496] This embodiment relates to the handling of calibration data used to detect the direction of a user's gaze. The detection of the gaze direction can be used, for example, to set a focus detection area in a camera, identify a main subject, and operate a GUI by gaze.

[0497] Note that the gaze detection method and the calibration method for gaze detection are publicly known, as described in, for example, Japanese Patent Application Laid-Open No. 2002-303781, and therefore detailed description thereof will be omitted. Here, it is assumed that camera body 1 has a peer-type finder capable of gaze detection and a calibration function.

[0498] 26(a) is a flowchart relating to the calibration process in this embodiment. For example, when the operation mode of the camera body 1 is changed to the calibration mode by operating an operation member of the camera body 1, the overall control CPU 101 starts the process shown in FIG. 26(a).

[0499] In the first and second embodiments, a calibrator 850 separate from the camera body 1 was required, but in this embodiment, there is no need to consider the direction of the face, and therefore calibration of gaze detection can be performed only by the camera body 1. However, this embodiment is common to the first and second embodiments in that a calibrator 850 is used to identify the calibration data to be used for the current user.

[0500] In S60001, the overall control CPU 101 receives the unique ID of the user's calibrator 850. This process is the same as the process described in the first embodiment using the flowchart in Fig. 24B, and therefore a description thereof will be omitted.

[0501] In step S60002, the overall control CPU 101 executes calibration processing related to gaze detection. Since the calibration processing can use a known method, such as the method described in the above-mentioned patent document, detailed description thereof will be omitted. Through the calibration processing, a correction value specific to the user is generated as calibration data to be used for the gaze detection function.

[0502] In S60003, the overall control CPU 101 determines whether or not the unique ID of the calibrator 850 was received in S60001, and if it is determined that it was received, executes S60004, and if not, executes S60005.

[0503] In step S60004, the overall control CPU 101 stores the calibration data in the built-in nonvolatile memory 102 of the camera body 1 in association with the unique ID of the calibrator and user information.

[0504] In S60005, the overall control CPU 101 stores the calibration data in association with user information in the built-in nonvolatile memory 102 of the camera body 1. Note that in S60004 and S60005, association with user information is not essential.

[0505] When the camera body 1 receives unique IDs from multiple calibrators 850, the overall control CPU 101 determines the unique ID to associate with the calibration data. As in the first embodiment, the overall control CPU 101 can determine the unique ID with the highest priority based on the received radio wave strength RSSI, information about the application running in the foreground, and the like, as the unique ID to associate with the calibration data.

[0506] Next, the operation of the camera body 1 in the gaze detection mode, which sets the focus detection area according to the gaze position detected using the gaze detection function, will be described using the flowchart shown in Fig. 26(b). For example, when the power-on operation using the power switch 11 of the camera body 1 is detected, the overall control CPU 101 starts the process shown in Fig. 26(b).

[0507] In S61001, the overall control CPU 101 receives a unique ID from the calibrator 850 present within the communication range, similar to S60001. Note that if the unique ID cannot be received even after a certain period of time has elapsed, for example, the overall control CPU 101 executes S61005.

[0508] In S61002, the overall control CPU 101 reads out the calibration data associated with the received unique ID from the built-in non-volatile memory 102. Again, if the camera body 1 receives unique IDs from multiple calibrators 850, the overall control CPU 101 can assign priorities to the unique IDs and read out the calibration data associated with the unique ID with the highest priority.

[0509] In S61003, the overall control CPU 101 determines whether or not the readout in S61002 was successful, and if it is determined to be successful, executes S61004, and if not, executes S61005.

[0510] In S61004, the overall control CPU 101 determines that the calibration data read out in S61002 is to be used to correct the line of sight detection result.

[0511] In S61005, the overall control CPU 101 determines to use default calibration data stored in advance in the built-in nonvolatile memory 102 to correct the gaze detection result. Alternatively, the overall control CPU 101 may determine the correction amount to be 0, or may determine to use the most recently used calibration data.

[0512] As described above, the present invention can also be applied to specifying calibration data related to gaze detection.

[0513] As explained in the first embodiment, calibration data may be stored in the calibrator 850, and in the gaze detection mode, the calibration data may be received from the calibrator 850. Furthermore, if there is no calibration data associated with the unique ID in S61003, the overall control CPU 101 may display that fact on the display member of the camera body 1, and prompt the user to perform calibration.

[0514] The gaze (gaze position) detected in gaze detection mode can also be used for processes other than setting the focus detection area. For example, it can be used to identify areas to apply processing to, such as the area for white balance (WB) correction, the area for metering for automatic exposure (AE), or the area for enlarging live view. It can also be used to select GUI parts (items, buttons, switches, etc.) displayed on the screen.

[0515] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0516] The disclosure of this embodiment includes the following imaging device, a control method thereof, and a program. (Item 1) An imaging device having a function of detecting a user's line of sight, a storage means for storing calibration data for correcting individual differences among users with respect to the detection result of the gaze direction; a receiving means for receiving identification information of an external device from the external device present within a communication range; an acquisition means for acquiring the calibration data associated with the identification information from the storage means; a correction means for correcting the detection result of the gaze direction using the acquired calibration data; An imaging device comprising: (Item 2) further comprising an assignment means for assigning priorities to the received identification information when the receiving means receives the identification information from a plurality of the external devices; the acquiring means acquires the calibration data associated with the identification information having the highest priority. 2. The imaging device according to item 1, (Item 3) 3. The imaging device according to item 2, wherein the allocation means allocates the priority to the plurality of pieces of identification information based on the type of application running on the external device that received the identification information. (Item 4) The imaging device described in item 3 is characterized in that the allocation means assigns a higher priority to the identification information of an external device that is executing an application related to a collaborative operation with the imaging device, among the plurality of external devices that have received the identification information, than to the identification information of an external device that is not executing an application related to a collaborative operation with the imaging device. (Item 5) The imaging device described in item 4, characterized in that the assignment means assigns a higher priority to the identification information of the external device that is running an application related to a collaborative operation with the imaging device, the closer the distance between the external device and the imaging device. (Item 6) The imaging device described in any one of items 3 to 5, characterized in that, when there is no external device running an application related to collaborative operation with the imaging device, the assignment means assigns a higher priority to the identification information of the external device that does not have a running application than to the identification information of the external device that is running an application that is not related to collaborative operation with the imaging device. (Item 7) The imaging device described in item 6, characterized in that the assignment means assigns a higher priority to the identification information of the external device that has no running application, the closer the distance to the imaging device is to the identification information of the external device. (Item 8) the calibration data is associated with distances to feature points of the user's head measured from an image captured by the imaging device in a calibration process executed at the time of generation; a measuring means for performing an image capture using the imaging device and measuring the distance from the obtained image when a plurality of pieces of calibration data associated with the same identification information exist in the storage means; further comprising an assignment means for assigning a priority to the calibration data associated with the identification information received by the receiving means; the assigning means assigns the priority based on a difference between the distance associated with the calibration data associated with the identification information received by the receiving means and the distance measured by the measuring means. 2. The imaging device according to item 1, (Item 9) 9. The imaging device according to any one of items 1 to 8, wherein the acquisition unit acquires the calibration data from the external device having the identification information when the calibration data associated with the identification information does not exist in the storage unit. (Item 10) 9. The imaging device according to any one of items 1 to 8, wherein the acquisition means acquires the calibration data from the external device having the identification information when the calibration data present in the external device having the identification information is newer than the calibration data present in the storage means and associated with the identification information. (Item 11) 11. The imaging device according to any one of items 1 to 10, wherein the imaging device is a wearable device, and the calibration data is used to correct a difference between the line of sight of the user wearing the imaging device and the center position of an area to be cut out from an image captured by the imaging device. (Item 12) 11. The imaging device according to any one of items 1 to 10, characterized in that the imaging device has a gaze detection function, and the calibration data is used to correct the detection result of the gaze direction of the user of the imaging device by the gaze detection function. (Item 13) A control method executed by an imaging device having a function of detecting a user's line of sight, comprising: receiving, from an external device within communication range, identification information of the external device; acquiring calibration data associated with the identification information from a storage means that stores calibration data for correcting individual differences among users with respect to the detection result of the gaze direction; correcting the gaze direction detection result using the acquired calibration data; A control method comprising: (Item 14) A program for causing a computer included in an imaging device having a function for detecting a user's line of sight to function as each of the means included in the imaging device described in any one of items 1 to 12.

[0517] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0518] 1... camera body, 10... imaging and detection unit, 101... overall control CPU, 800... display device, 801... display device control unit, 805... in-camera, 806... face sensor, 807... angular velocity sensor, 808... acceleration sensor, 850... calibrator

Claims

1. An imaging device having a function of detecting a user's line of sight, a storage means for storing calibration data for correcting individual differences among users with respect to the detection result of the gaze direction; a receiving means for receiving identification information of an external device from the external device present within a communication range; an acquisition means for acquiring the calibration data associated with the identification information from the storage means; a correction means for correcting the detection result of the gaze direction using the acquired calibration data; An imaging device comprising:

2. further comprising an assignment means for assigning priorities to the received identification information when the receiving means receives the identification information from a plurality of the external devices; the acquiring means acquires the calibration data associated with the identification information having the highest priority.

2. The imaging device according to claim 1.

3. 3. The imaging device according to claim 2, wherein the allocation unit allocates the priority to the plurality of pieces of identification information based on the type of application currently being executed in the external device that received the identification information.

4. The imaging device according to claim 3, characterized in that the allocation means assigns a higher priority to the identification information of an external device, among the plurality of external devices that have received the identification information, that is currently executing an application related to collaborative operation with the imaging device, than to the identification information of an external device that is not currently executing an application related to collaborative operation with the imaging device.

5. The imaging device according to claim 4, characterized in that the assignment means assigns a higher priority to the identification information of the external device that is running an application related to collaborative operation with the imaging device, the closer the distance between the external device and the imaging device.

6. The imaging device described in claim 3, characterized in that, when there is no external device running an application related to collaborative operation with the imaging device, the assignment means assigns a higher priority to the identification information of the external device that does not have a running application than to the identification information of the external device that is running an application that is not related to collaborative operation with the imaging device.

7. 7. The imaging device according to claim 6, wherein the allocation means assigns a higher priority to the identification information of the external device that has no running application, the closer the distance from the imaging device to the identification information of the external device.

8. the calibration data is associated with distances to feature points of the user's head measured from an image captured by the imaging device in a calibration process executed at the time of generation; a measuring means for performing an image capture using the imaging device and measuring the distance from the obtained image when a plurality of pieces of calibration data associated with the same identification information exist in the storage means; further comprising an assignment means for assigning a priority to the calibration data associated with the identification information received by the receiving means; the assigning means assigns the priority based on a difference between the distance associated with the calibration data associated with the identification information received by the receiving means and the distance measured by the measuring means.

2. The imaging device according to claim 1.

9. 2. The imaging device according to claim 1, wherein, when the calibration data associated with the identification information does not exist in the storage unit, the acquisition unit acquires the calibration data from the external device having the identification information.

10. The imaging device according to claim 1, characterized in that, when calibration data that is newer than the calibration data associated with the identification information and that is stored in the storage means is stored in the external device having the identification information, the acquisition means acquires the calibration data from the external device having the identification information.

11. The imaging device according to claim 1, characterized in that the imaging device is a wearable device, and the calibration data is used to correct a difference between the line of sight of the user wearing the imaging device and the center position of an area to be cut out from an image captured by the imaging device.

12. 2. The imaging device according to claim 1, wherein the imaging device has a gaze detection function, and the calibration data is used to correct the detection result of the gaze direction of the user of the imaging device by the gaze detection function.

13. A control method executed by an imaging device having a function of detecting a user's line of sight, comprising: receiving, from an external device within communication range, identification information of the external device; acquiring calibration data associated with the identification information from a storage means that stores calibration data for correcting individual differences among users with respect to the detection result of the gaze direction; correcting the gaze direction detection result using the acquired calibration data; A control method comprising:

14. 13. A program for causing a computer included in an imaging device having a function of detecting a user's line of sight to function as each of the means included in the imaging device according to claim 1.

Images