Imaging apparatus, portable device, and calibrator, and control method and storage medium therefor

Abstract

The present application relates to a camera device, a portable device, a calibrator, a control method thereof, and a storage medium. A wearable camera device eliminates manual changes in a camera direction during image capturing, and can easily obtain images recording an experience while focusing attention on the experience. The camera device includes a viewing direction detection unit, a camera unit, and an image output unit. The viewing direction detection unit is worn on a body of a user other than a head, and detects a viewing direction of the user. The camera unit is worn on the body of the user and captures an image. The image output unit outputs an image corresponding to the viewing direction based on the image captured by the camera unit.

Description

TECHNICAL FIELD

[0001] The present application relates to a wearable camera device, a portable device and a calibrator that communicate with the camera device, a control method thereof, and a storage medium storing a control program thereof, and particularly to a camera device used as a motion camera, a portable device, a calibrator, a control method thereof, and a storage medium storing a control program thereof. BACKGROUND

[0002] In a case where a user takes an image of a subject with a camera, the user needs to continuously direct the camera toward the subject. Therefore, since the user is busy with the image-taking operation, the user has difficulty in managing actions other than the image-taking action. Also, since the user focuses attention on the image-taking operation, the user has difficulty in focusing attention on the user's experience at the site.

[0003] For example, regarding the image-taking operation, a parent as a user cannot play with a child as a subject during the image-taking operation, and the image-taking operation becomes impossible while playing with the child.

[0004] Further, regarding the focusing of attention, when a user takes an image while watching a sports game, the user cannot cheer or cannot remember the contents of the game, and the image-taking operation becomes impossible while focusing attention on watching the sports game. Similarly, when a user takes an image during a trip with a group, the user cannot experience the same level of impression as other members, and the image-taking is neglected when prioritizing the experience.

[0005] As a method for solving these problems, there is a method for fixing a camera to a head using a head-mounted accessory to take an image in an observation direction. This enables a user to take an image without having to be busy with the image-taking operation. Further, there is also a method for taking an image in a wide area during an experience using a full-sphere camera. This enables a user to focus attention on the experience. After the experience, the user extracts a desired image portion from the taken full-sphere image and edits the image portion to leave an image of the experience.

[0006] However, the former method requires troublesome actions of equipping the head with the head-mounted accessory 902 having the body of the motion camera 901 fixed thereto as shown in Figure 63A Further, as shown in Figure 63B in a case where a user equips the head with the motion camera 901 using the head-mounted accessory 902, the appearance is poor, and also makes the user's hairstyle messy. Further, the user feels uneasy due to the weight of the head-mounted accessory 902 and the motion camera 901 and the bad appearance to a third person, for the existence of the head-mounted accessory 902 and the motion camera 901. Therefore, since the user cannot focus attention on the experience in the state shown in Figure 63B Therefore, the present application has been made to solve the above-mentioned problems, and an object of the present application is to provide a wearable camera device, a portable device and a calibrator that communicate with the camera device, a control method thereof, and a storage medium storing a control program thereof, which enable a user to take an image without having to be busy with the image-taking operation, and to focus attention on an experience at a site. Figure 63B The style shown feels resistant, so the user can have difficulty taking an image.

[0007] On the other hand, the latter method requires a series of operations such as image conversion and extraction position designation. For example, there is known a technique such as Figure 64 A full-sky camera 903 equipped with a lens 904 and a photographing button 905 is shown. The lens 904 is one of a pair of fisheye lenses provided on both sides of the housing of the full-sky camera 903 for taking a half-sky image. The full-sky camera 903 takes a full-sky image using these fisheye lenses. Then, a full-sky image is obtained by synthesizing images taken using the pair of fisheye lenses.

[0008] Figure 65A 、 Figure 65B and Figure 65C are diagrams showing an example of conversion processing of an image taken by the full-sky camera 903.

[0009] Figure 65A An example of a full-sky image obtained by the full-sky camera 903 is shown, and includes a user 906, a child 907, and a tree 908 as subjects. Since the image is a full-sky image obtained by synthesizing projection images of a pair of fisheye lenses, the user 906 is greatly distorted. Further, since a body part of the child 907, which is a subject that the user 906 wants to take, is located at a peripheral portion of a photographing region of the lens 904, the body part is greatly distorted to the left and right and stretched. On the other hand, since the tree 908 is a subject located in front of the lens 904, the tree 908 is taken in a state without great distortion.

[0010] In order to generate an image of a visual field that people generally look at from the image shown in Figure 65A , it is necessary to extract a part of the image, perform planar conversion, and display.

[0011] Figure 65B is an image located in front of the lens 904 extracted from the image shown in Figure 65A . In the image of Figure 65B , the tree 908 is shown in the center of a visual field that people generally look at. However, since the image of Figure 65B does not include the child 907, which is a subject that the user 906 wants to take, the user must change the extraction position. Specifically, in this case, it is necessary to move the extraction position 30° to the left and down in Figure 65A . Figure 65C A display image obtained by extracting the moved position and performing planar conversion is shown. In this way, in order to obtain an image of a visual field that the user wants to take from the image of Figure 65A , it is necessary to move the extraction position 30° to the left and down in Figure 65CIn order to extract the image of the desired region from the image of the entire region, the user must extract the desired region and must perform a plane conversion. Therefore, although the user can concentrate on the experience during the experience (during the photographing), the subsequent workload becomes enormous.

[0012] Japanese Patent Application Publication No. 2007-74033 (JP 2007-74033 A) discloses a technique that uses a second camera that photographs a user in addition to a first camera that photographs a subject. The technique calculates a moving direction and a line-of-sight direction of the user from an image photographed by the second camera, determines a photographing direction of the first camera, and photographs an image of the subject estimated based on the taste and the state of the user.

[0013] Further, Japanese Patent Application Publication No. 2017-60078 (JP 2017-60078 A) (U.S. Patent Application 20170085841) discloses an image recording system that includes a sensor device attached to the head of a user and a photographing apparatus separately attached to the body or a bag of the user. The sensor device is composed of a gyro sensor or an acceleration sensor, and detects the viewing direction of the user. The photographing apparatus photographs an image in the viewing direction detected by the sensor device.

[0014] However, since the second camera of JP 2007-74033 A photographs the user from a position far from the user, the second camera needs high optical performance to calculate the moving direction and the line-of-sight direction of the user from the image photographed by the second camera. Further, since high operation processing capability is needed to process the image photographed by the second camera, the scale of the device becomes large. Further, even if the high optical performance and the high operation processing capability are satisfied, the viewing direction of the user cannot be accurately calculated. Therefore, since the subject that the user wants to photograph cannot be estimated with sufficient accuracy based on the taste and the state of the user, an image other than the image that the user wants can be photographed.

[0015] Further, since the sensor device of JP 2017-60078 A directly detects the viewing direction of the user, the user needs to equip the sensor device to the head, which cannot solve the troubles when any device is attached to the head as described above. Further, in the case where the sensor device is composed of a gyro sensor or an acceleration sensor, a certain degree of accuracy can be obtained in the detection of the relative viewing direction. However, since the detection accuracy of the absolute viewing direction (particularly, the horizontal rotation direction) cannot be obtained, there is a problem in practical use. SUMMARY

[0016] The present application provides a wearable camera device, a portable device and a calibrator for communicating with the camera device, a control method thereof, and a storage medium for storing a control program thereof, which eliminates manual changing of a camera direction during image capturing, and can easily obtain images recording an experience while focusing attention on the experience.

[0017] Therefore, an aspect of the present application provides a camera device including: an observation direction detection unit worn on a body of a user other than a head, and configured to detect an observation direction of the user; a camera unit worn on the body of the user, and configured to capture an image; and an image output unit configured to output an image corresponding to the observation direction based on the image captured by the camera unit.

[0018] An aspect of the present application provides a portable device for wirelessly connecting to a camera device, the portable device including: a video file receiving unit configured to receive a video file in which metadata and an image of each frame are encoded for each frame, the metadata including in-image position information showing a position and a size of an image corresponding to an observation direction of a user with respect to the image of each frame captured by the camera device; a first extraction unit configured to extract the metadata from the video file; a second extraction unit configured to extract the image of the frame encoded together with the metadata extracted from the video file; a frame image correction unit configured to correct the image of the frame extracted by the second extraction unit using the metadata extracted by the first extraction unit; and a video recording unit configured to record the image of the frame corrected by the frame image correction unit as a video image.

[0019] An aspect of the present application provides a calibrator for wirelessly connecting to a camera device, the calibrator including: a first display unit configured to display a positioning indicator captured by a camera unit of the camera device during calibration; a face detection unit configured to detect a face of a user wearing the camera device on his / her body; and a second display unit configured to display a button that can be pressed to transmit an instruction of the calibration to the camera device in a case where it is judged from a detection result of the face detection unit that the user is looking at the positioning indicator.

[0020] An aspect of the present application provides a control method for a camera device, the control method including: detecting an observation direction of a user with an observation direction detection unit worn on a body of the user other than a head; capturing an image with a camera unit worn on the body of the user; and outputting an image corresponding to the observation direction based on the captured image.

[0021] An aspect of the present application provides a control method for a portable device for wirelessly connecting to a camera device, the control method including receiving a video file in which metadata and images of frames are encoded for each frame, the metadata including in-image position information showing a position and a size of an image corresponding to a user's viewing direction with respect to the images of the frames taken by the camera device, extracting the metadata from the video file, extracting the images of the frames encoded together with the metadata extracted from the video file, correcting the extracted images of the frames using the extracted metadata, and recording the corrected images of the frames as video images.

[0022] An aspect of the present application provides a control method for a calibrator for wirelessly connecting to a camera device, the control method including displaying a positioning marker taken by an imaging unit of the camera device during calibration, detecting a face of a user who wears the camera device on his own body, and displaying a button that can be pressed to send an instruction of the calibration to the camera device in a case where it is determined from a detection result that the user is looking at the positioning marker.

[0023] An aspect of the present application provides a non-transitory computer readable storage medium for storing a control program that causes a computer to execute a control method for a camera device, the control method including detecting a viewing direction of a user with a viewing direction detection unit worn on a body of the user other than a head, taking an image with an imaging unit worn on the body of the user, and outputting an image corresponding to the viewing direction based on the taken image.

[0024] An aspect of the present application provides a non-transitory computer readable storage medium for storing a control program that causes a computer to execute a control method for a portable device for wirelessly connecting to a camera device, the control method including receiving a video file in which metadata and images of frames are encoded for each frame, the metadata including in-image position information showing a position and a size of an image corresponding to a user's viewing direction with respect to the images of the frames taken by the camera device, extracting the metadata from the video file, extracting the images of the frames encoded together with the metadata extracted from the video file, correcting the extracted images of the frames using the extracted metadata, and recording the corrected images of the frames as video images.

[0025] An aspect of the present application provides a non-transitory computer-readable storage medium for storing a control program that causes a computer to execute a control method for a calibrator for wirelessly connecting to an imaging device, the control method including: displaying a positioning indicator captured by an imaging unit of the imaging device during calibration; detecting a face of a user who wears the imaging device on his or her body; and displaying a button that can be pressed to send an instruction of the calibration to the imaging device in a case where it is determined from a detection result that the user is looking at the positioning indicator.

[0026] According to the present application, manual change of the imaging direction during image capturing becomes unnecessary, and images recording the experience can be easily obtained while focusing attention on the experience.

[0027] Further features of the present application will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1A is an appearance view showing a camera body including an imaging / detecting unit as the imaging device according to the first embodiment.

[0029] Figure 1B is a view showing a state in which a user wears the camera body.

[0030] Figure 1C is a view showing a battery unit in the camera body seen from the rear side. Figure 1A

[0031] Figure 1D is an appearance view showing a display device as the portable device according to the first embodiment, which is separate from the camera body.

[0032] Figure 2A is a front view showing the imaging / detecting unit in the camera body.

[0033] Figure 2B is a view showing a shape of a belt portion of a connecting member in the camera body.

[0034] Figure 2C is a rear view showing the imaging / detecting unit.

[0035] Figure 2D is a top view showing the imaging / detecting unit.

[0036] Figure 2E is a view showing a structure of a face direction detecting unit arranged inside the imaging / detecting unit and below a face direction detecting window in the camera body.

[0037] Figure 2F ​is a diagram showing a state in which the user wears the camera body as seen from the left side of the user.

[0038] Figure 3A , Figure 3B and Figure 3C is a diagram showing details of the battery unit.

[0039] Figure 4 is a functional block diagram showing the camera body according to the first embodiment.

[0040] Figure 5 is a block diagram showing a hardware structure of the camera body according to the first embodiment.

[0041] Figure 6 is a block diagram showing a hardware structure of the display device according to the first embodiment.

[0042] Figure 7A is a flowchart schematically showing an imaging / recording process according to the first embodiment, which is executed by the camera body and the display device.

[0043] Figure 7B is a flowchart showing a subroutine of the preparation process in step S100 of Figure 7A according to the first embodiment.

[0044] Figure 7C is a flowchart showing a subroutine of the face direction detection process in step S200 of Figure 7A according to the first embodiment.

[0045] Figure 7D is a flowchart showing a subroutine of the recording direction / region determination process in step S300 of Figure 7A according to the first embodiment.

[0046] Figure 7E is a flowchart showing a subroutine of the recording region visualization process in step S500 of Figure 7A according to the first embodiment.

[0047] Figure 7F is a diagram for explaining the processes in steps S200 to S600 of Figure 7A according to the first embodiment.

[0048] Figure 8A is a diagram showing an image of the user as seen from the face direction detection window.

[0049] Figure 8B is a diagram showing a case where a fluorescent lamp in the room appears as a background in the image of the user as seen from the face direction detection window.

[0050] Figure 8Cis a diagram showing an image obtained by imaging the user and the fluorescent lamp as a background through the face direction detection window onto the sensor of the infrared detection device in a state where the infrared LED is not lit. Figure 8B

[0051] Figure 8D is a diagram showing an image obtained by imaging the user and the fluorescent lamp as a background through the face direction detection window onto the sensor of the infrared detection device in a state where the infrared LED is lit. Figure 8B

[0052] Figure 8E is a diagram showing a difference image calculated by subtracting the image of Figure 8D from the image of Figure 8C .

[0053] Figure 8F is a diagram showing a result obtained by adjusting the gradation of the difference image in Figure 8E to fit the scale of the light intensity of the reflection component of the infrared light projected to the face and the neck of the user.

[0054] Figure 8G is a diagram obtained by superimposing on Figure 8F the attached reference numerals indicating the parts of the body of the user, the double circle showing the position of the throat, and the black circle showing the position of the chin.

[0055] Figure 8H is a diagram showing a difference image calculated by a method similar to Figure 8E when the face of the user is directed to the right.

[0056] Figure 8I is a diagram showing a result obtained by adjusting the gradation of the difference image in Figure 8H to fit the scale of the light intensity of the reflection component of the infrared light projected to the face and the neck of the user, and superimposing the double circle showing the position of the throat and the black circle showing the position of the chin.

[0057] Figure 8J is a diagram showing an image of the user with the face directed upward by 33° as seen from the face direction detection window.

[0058] Figure 8K is a diagram showing a result obtained by adjusting the gradation of a difference image calculated by a method similar to Figure 8E when the user directs the face upward by 33° to fit the scale of the light intensity of the reflection component of the infrared light projected to the face and the neck of the user, and superimposing the double circle showing the position of the throat and the black circle showing the position of the chin. ​​

[0059] Figure 9 is a timing chart showing the lighting timing of the infrared LED and the related signals.

[0060] 10A to 10D is a diagram illustrating movement of the user's face in the vertical direction.

[0061] Figure 11A is a diagram illustrating a target field of view set in a super-wide-angle image captured by the imaging unit of the camera body in a case where the user is facing straight ahead.

[0062] Figure 11B is a diagram illustrating an image in the target field of view extracted from the super-wide-angle image of Figure 11A .

[0063] Figure 11C is a diagram illustrating a target field of view set in a super-wide-angle image in a case where the user is observing the A subject.

[0064] Figure 11D is a diagram illustrating an image obtained by correcting distortion and blurring of the image in the target field of view extracted from the super-wide-angle image of Figure 11C .

[0065] Figure 11E is a diagram illustrating a target field of view set in a super-wide-angle image in a case where the user is observing the A subject at a field-angle set value smaller than the field-angle set value in Figure 11C .

[0066] Figure 11F is a diagram illustrating an image obtained by correcting distortion and blurring of the image in the target field of view extracted from the super-wide-angle image of Figure 11E .

[0067] Figure 12A is a diagram illustrating an example of a target field of view set in a super-wide-angle image.

[0068] Figure 12B is a diagram illustrating an example of a target field of view set in a super-wide-angle image in a case where the field-angle set value is the same as the field-angle set value of the target field of view of Figure 12A and the observation direction is different.

[0069] Figure 12C is a diagram illustrating another example of a target field of view set in a super-wide-angle image in a case where the field-angle set value is the same as the field-angle set value of the target field of view of Figure 12A and the observation direction is different.

[0070] Figure 12D is a diagram illustrating a case where the observation direction is different from the target field of view of Figure 12CFIG. 1 is a diagram showing an example of a target field of view set in an ultra-wide-angle image in a case where the observation direction of the target field of view is the same and the field angle setting value is smaller.

[0071] Figure 12E It is shown in Figure 12A A diagram showing an example of an image stabilization margin assigned around a target field of view corresponding to a predetermined image stabilization level.

[0072] Figure 12F It is shown in Figure 12B The target field of view shown is given around Figure 12E A graph of an example of an image stabilization margin corresponding to the same image stabilization level.

[0073] Figure 12G It is shown in Figure 12D The target field of view shown is given around Figure 12E A graph of an example of an image stabilization margin corresponding to the same image stabilization level.

[0074] Figure 13 : is a diagram showing a menu screen for setting various setting values ​​of a video image mode, which is displayed on the display unit of the display device before an image capturing operation of the camera body.

[0075] Figure 14 It shows Figure 7A Flowchart of the subroutine of the master recording process in step S600.

[0076] Figure 15 This is a diagram showing the data structure of an image file generated by the master recording process.

[0077] Figure 16 yes Figure 7A Flowchart of a subroutine of the transmission process to the display device in step S700.

[0078] Figure 17 It shows Figure 7A Flowchart of a subroutine of the optical correction process in step S800.

[0079] 18A to 18F Is used to illustrate Figure 17 FIG. 10 is a diagram of the process of applying distortion correction in step S803.

[0080] Figure 19 It shows Figure 7A Flowchart of a subroutine of the image stabilization process in step S900.

[0081] Figure 20A and Figure 20B is a diagram showing details of a calibrator used in the calibration process according to the second embodiment.

[0082] Figure 21 is a flowchart showing a calibration process according to the second embodiment performed by the camera body and the calibrator.

[0083] Figure 22A is a diagram showing a screen displayed on the display unit of the calibrator in step S3103 in Figure 21

[0084] Figure 22B is a diagram showing the user holding the calibrator in a state in which the calibrator is shown by the indication display as Figure 22A

[0085] Figure 22C is a diagram showing the entire super-wide-angle image captured by the imaging lens in a state of Figure 22B

[0086] Figure 22D is a diagram showing an image obtained by correcting aberration of the super-wide-angle image shown in Figure 22C

[0087] Figure 22E is a diagram showing a face direction image obtained by the face direction detection unit in step S3108 in Figure 21

[0088] Figure 22F is a diagram showing the built-in camera image displayed in step S3107 in Figure 21

[0089] Figure 23A is a diagram showing a screen displayed on the display unit of the calibrator in step S3103 in Figure 21

[0090] Figure 23B is a diagram showing the user holding the calibrator in a state in which the calibrator is shown by the indication display as Figure 23A

[0091] Figure 23C is a diagram showing the entire super-wide-angle image captured by the imaging lens in a state of Figure 23B

[0092] Figure 23D is a diagram showing an image obtained by correcting aberration of the super-wide-angle image shown in Figure 23C

[0093] Figure 23E ​​​​​​​​​​is a diagram showing the calibration operation during the upper right direction of the user. Figure 21 Schematic diagram of the facial direction image obtained by the facial direction detection unit in step S3108 in .

[0094] Figure 24A 、 Figure 24B and Figure 24C 1 and 2 are diagrams for explaining delayed extraction of an image in the third embodiment.

[0095] Figure 25A and Figure 25B is a diagram showing the trajectory of facial movement maintained in the third embodiment.

[0096] Figure 26 : is a flowchart showing the visually induced motion sickness prevention process according to the third embodiment.

[0097] Figures 27A to 27F Graphs for explaining extraction region correction processing according to the fourth embodiment.

[0098] Figure 28A is a flowchart illustrating recording direction / area determination processing according to the fourth embodiment. Figure 28B It shows Figure 28A Flowchart of the extraction area correction process in step 400.

[0099] Figure 29A and Figure 29B 1 is a schematic diagram for explaining the relationship between the user's field of view and the target field of view when a close-range subject is an observation target in the first embodiment.

[0100] Figure 30 is a diagram showing the external appearance of a camera body including an image pickup apparatus according to a fifth embodiment.

[0101] Figure 31 is a block diagram showing the hardware configuration of a camera body according to the fifth embodiment.

[0102] Figure 32A and Figure 32B is a schematic diagram for explaining the relationship between the user, the calibrator, and the target field of view during calibration processing including parallax correction mode processing in the fifth embodiment.

[0103] Figure 33A is shown as a diagram according to the fifth embodiment Figure 7A Flowchart of the parallax correction mode process which is a part of the preparation process in step S100.

[0104] Figure 33B is a diagram showing a method according to the fifth embodiment Figure 7Aa flowchart of a subroutine of the recording direction / region determination process in step S300.

[0105] Figure 34A 、 Figure 34B and Figure 34C is a diagram showing the relationship between the defocus map generated in step S5302 of Figure 33B

[0106] Figure 35 is a flowchart showing the observation direction determination process according to the sixth embodiment.

[0107] Figure 36A is a diagram showing the relationship between the observation direction detection state of the user for each frame and the captured image in the subject loss mode according to the sixth embodiment.

[0108] Figure 36B is a diagram showing the relationship between the observation direction detection state of the user for each frame and the captured image in the subject loss mode according to the sixth embodiment.

[0109] Figure 37A 、 Figure 37B and Figure 37C are diagrams for explaining the relationship between the observation direction and the face region available for the detection of the observation direction according to the seventh embodiment.

[0110] Figure 38 is a flowchart showing the observation direction determination process at the time of obtaining the face direction according to the seventh embodiment, which is executed instead of the process in step S6004 of Figure 35

[0111] Figure 39 is a diagram showing the relationship between the face direction and the face direction reliability in the seventh embodiment.

[0112] Figure 40 is a diagram showing the observation direction determination process at the time of obtaining the face direction in the seventh embodiment.

[0113] Figure 41A 、 Figure 41B and Figure 41C are enlarged side views of the imaging / detection unit.

[0114] Figure 42A 、 Figure 42B and Figure 42C are side views showing the state in which the user wears the camera body.

[0115] Figure 43A 、 Figure 43B and Figure 43C are enlarged side views of the imaging / detection unit in a state in which the connecting member is not shown.​​

[0116] Figure 44A 、 Figure 44B and Figure 44C is a side view showing a state in which a user wears the camera body in a state in which the connecting member is not shown.

[0117] Figures 45A to 45G is a diagram showing various combinations of the band portion and a connecting surface of a cross section of the cable that is integral with the band portion.

[0118] Figure 46A is a block diagram showing a hardware structure of a display device connected to the camera body including the imaging device according to the ninth embodiment.

[0119] Figure 46B is a functional block diagram showing the camera body according to the ninth embodiment.

[0120] Figure 47 is a functional block diagram showing the camera body and the display device according to the tenth embodiment.

[0121] Figure 48 is a flowchart schematically showing an imaging / recording process according to the tenth embodiment, which is executed by the camera body and the display device.

[0122] Figure 49 is a functional block diagram showing the camera body and the display device according to the eleventh embodiment.

[0123] Figure 50 is a flowchart schematically showing an imaging / recording process according to the eleventh embodiment, which is executed by the camera body and the display device.

[0124] Figure 51A is an appearance view of the camera body according to the twelfth embodiment.

[0125] Figure 51B is a perspective view showing details of an imaging / detecting unit that is a part of the camera body according to the twelfth embodiment.

[0126] Figure 51C is a perspective view showing a state in which the imaging unit of the imaging / detecting unit in Figure 51B is turned left by 30°. Figure 51D is a perspective view showing a state in which the imaging unit is turned down by 30°.

[0127] Figure 52 is a functional block diagram showing the camera body according to the twelfth embodiment.

[0128] Figure 53 is a block diagram showing a hardware structure of the camera body according to the twelfth embodiment.

[0129] Figure 54 Fig. 23 is a flowchart schematically showing an imaging / recording process according to the twelfth embodiment, which is executed by the camera body and the display device.

[0130] Figure 55 Fig. 24 is a flowchart showing a subroutine of the face direction detection process in step S12300 of Figure 54 Fig. 25 is a flowchart showing a subroutine of the visualization process in step S12500 of

[0131] Figure 56 Fig. 26 is a flowchart showing a subroutine of the face direction calculation process according to the twelfth embodiment. Figure 54

[0132] Figure 57 Fig. 27 is a block diagram showing a hardware structure of the camera body according to the thirteenth embodiment.

[0133] Figure 58A Figure 58B Fig. 28 is a schematic diagram showing an example of a learning image used in the thirteenth embodiment. Figure 58C

[0134] Figure 59 Fig. 29 is a flowchart showing a face direction detection process using machine learning according to the thirteenth embodiment.

[0135] Figure 60 Fig. 30 is a block diagram showing a hardware structure of the camera body according to the fourteenth embodiment.

[0136] Figure 61A Fig. 31 is a schematic diagram showing a distance image generated by a ToF (Time of Flight) device of the camera body according to the fourteenth embodiment, in a state where the ToF device is arranged at a clavicle position of a user and measures upward.

[0137] Figure 61B Fig. 32 is a schematic diagram showing an image in which a face portion is extracted by applying threshold processing to the distance image of Figure 61A Fig. 33 is a schematic diagram showing an image in which

[0138] Figure 61C Fig. 34 is a schematic diagram showing an image in which the image of Figure 61B is divided into regions according to distance information.

[0139] Figure 61D Fig. 35 is a schematic diagram showing an image in which a throat position and a chin position are extracted from the image of Figure 61C Fig. 36 is a flowchart showing a face direction calculation process according to the fourteenth embodiment.

[0140] Figure 62 Fig. 37 is a schematic diagram showing an example of a learning image used in the fifteenth embodiment.

[0141] Figure 63A Fig. 38 is a flowchart showing a face direction detection process according to the fifteenth embodiment.​​​ Figure 63B 1 is a diagram showing a structural example of a camera fixed to a head using a conventional head-fixing accessory.

[0142] Figure 64 1 is a diagram showing a configuration example of a conventional full celestial spherical camera.

[0143] Figure 65A 、 Figure 65B and Figure 65C It shows Figure 64 FIG. 1 is a diagram showing an example of conversion processing of an image captured by a full celestial camera. DETAILED DESCRIPTION

[0144] Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

[0145] Figures 1A to 1D This figure illustrates a camera system consisting of a camera body 1 and a display device 800 separate from the camera body 1. The camera body 1 includes an imaging / detection unit 10, which serves as a wearable imaging device according to the first embodiment. In this embodiment, the camera body 1 and the display device 800 are separate devices, but the two may be integrated.

[0146] Figure 1A 1 is a diagram showing the appearance of the camera body 1. Figure 1A As shown, the camera body 1 is provided with an imaging / detection unit 10, a battery unit (power supply unit) 90, a right side connection member 80R, and a left side connection member 80L. The right side connection member 80R is on the right side of the user's body ( Figure 1A The left side connecting member 80L is connected to the left side of the user's body ( Figure 1A The right side in the figure) connects the imaging / detection unit 10 and the battery unit 90.

[0147] The imaging / detection unit 10 is provided with a face direction detection window 13 , a start switch 14 , a stop switch 15 , an imaging lens 16 , an LED 17 , and microphones 19L and 19R.

[0148] The face direction detection window 13 allows the infrared LED 22 ( Figure 5 The infrared light projected by the infrared irradiation unit is transmitted to detect the position of the user's facial parts. The facial direction detection window 13 also allows the reflected infrared light from the face to pass through.

[0149] The start switch 14 is used to start the image pickup operation. The stop switch 15 is used to stop the image pickup operation. The image pickup lens 16 guides the light to be photographed to the solid-state image sensor 42 ( Figure 5). The LED 17 indicates a state in which the imaging operation is in progress or a warning.

[0150] The microphones 19R and 19L collect peripheral sounds. The microphone 19L collects sounds on the left side (right side in Figure 1A ) of the user's periphery. The microphone 19R collects sounds on the right side (left side in Figure 1A ) of the user's periphery.

[0151] Figure 1B is a view showing a state in which the user wears the camera body 1. In a case where the user wears the camera body 1 so that the battery unit 90 will reach the back side of the user and the imaging / detecting unit 10 will reach the front side of the user's body, the imaging / detecting unit 10 is supported in a state of being urged in a direction toward the chest by the left and right connecting members 80L and 80R connected to the left and right ends of the imaging / detecting unit 10, respectively. Thereby, the imaging / detecting unit 10 is positioned in front of the user's collarbone. At this time, the face direction detecting window 13 is positioned below the user's jaw. Inside the face direction detecting window 13, the infrared condenser lens 26 shown in Figure 2E is arranged. The optical axis (detection optical axis) of the infrared condenser lens 26 is directed toward the user's face and is directed in a direction different from the optical axis (imaging optical axis) of the imaging lens 16. The face direction detecting unit 20 (see Figure 5 ) including the infrared condenser lens 26 detects the user's viewing direction on the basis of the position of the face part. This enables the imaging unit 40 described later to image an object in the viewing direction.

[0152] Adjustment of the setting position due to individual differences in body shape and differences in clothing will be described later. Further, since the imaging / detecting unit 10 is arranged in the front side of the body and the battery unit 90 is arranged in the back side in this way, the weight of the camera body 1 is dispersed, which alleviates the user's fatigue and reduces displacement of the camera body 1 due to centrifugal force caused by the user's movement.

[0153] Although an example in which the user wears the camera body 1 in a manner that the imaging / detecting unit 10 will be positioned in front of the user's collarbone is described, the present embodiment is not limited to this example. That is, the user can wear the camera body 1 at any position on the user's body other than the head, as long as the camera body 1 can detect the user's viewing direction and the imaging unit 40 can image an object in the viewing direction.

[0154] Figure 1C is a view of the battery unit 90 seen from the back side in Figure 1A . In Figure 1C , the battery unit 90 is provided with a charging cable insertion slot 91, adjustment buttons 92L and 92R, and a spine-avoiding cutout 93.

[0155] A charging cable (not shown) can be connected to the charging cable insertion slot 91. The battery 94L and 94R (see Figure 3A ) are charged through the charging cable, and the camera / detection unit 10 is supplied with electric power through the charging cable.

[0156] The adjustment buttons 92L and 92R are used to adjust the lengths of the band portions 82L and 82R of the left and right connecting members 80L and 80R, respectively. The adjustment button 92L is used to adjust the left band portion 82L, and the adjustment button 92R is used to adjust the right band portion 82R. In the present embodiment, although the lengths of the band portions 82L and 82R are independently adjusted by the adjustment buttons 92L and 92R, the lengths of the band portions 82L and 82R can be simultaneously adjusted by one button.

[0157] The spine-avoiding cutout 93 is formed so as to avoid the user's spine, so that the battery unit 90 will not contact the spine. Since the spine-avoiding cutout 93 avoids the convex portion of the user's spine, the feeling of discomfort when worn is reduced, and lateral displacement of the battery unit 90 is prevented.

[0158] Figure 1D is an external view showing a display device 800 as a portable device according to the first embodiment, which is separate from the camera body 1. As Figure 1D indicated, the display device 800 is provided with an A button 802, a display unit 803, a B button 804, a built-in camera 805, a face sensor 806, an angular velocity sensor 807, and an acceleration sensor 808. Further, the display device 800 is provided with a wireless LAN unit (not shown in Figure 1D ) that enables high-speed connection with the camera body 1.

[0159] The A button 802 has the function of a power button of the display device 800. The display device 800 receives an ON / OFF operation by long pressing the A button 802, and receives designation of another processing timing by short pressing the A button 802.

[0160] The display unit 803 is used to check an image captured by the camera body 1, and can display a menu screen required for setting. In the present embodiment, a transparent touch sensor provided on the surface of the display unit 803 receives a touch operation on a screen (e.g., a menu screen) being displayed.

[0161] The B button 804 functions as a calibration button 854 used for a calibration process described later.

[0162] The built-in camera 805 can capture an image of a person who is observing the display device 800.

[0163] The face sensor 806 detects the face shape and the viewing direction of a person who is viewing the display device 800. The detailed structure of the face sensor 806 is not limited. For example, a structured light sensor, a ToF (Time of Flight) sensor, and a millimeter wave radar can be employed.

[0164] Since the angular velocity sensor 807 is built in the display device 800, the meaning of the angular velocity sensor 807 as a see-through view is shown by a broken line. Since the display device 800 of the present embodiment is also provided with the function of a calibrator described later, a three-axis gyro sensor that enables detection in the X, Y, and Z directions is provided.

[0165] The acceleration sensor 808 detects the posture of the display device 800. Note that a general smart phone is employed as the display device 800 according to the present embodiment. The camera system of the present embodiment is realized by adjusting the firmware in the smart phone to the firmware of the camera body 1. On the other hand, the camera system of the present embodiment can be realized by adjusting the firmware of the camera body 1 to the application and the OS of the smart phone as the display device 800.

[0166] Figures 2A to 2F is a view that explains the camera / detection unit 10 in detail. In the view from the Figure 2A , components having the same function as the already explained parts are denoted by the same reference numerals, and the explanation of the components in the present specification is omitted.

[0167] Figure 2A is a front view that shows the camera / detection unit 10. The right side connecting member 80R has a belt portion 82R and an angle holding member 81R of a hard material for holding the angle with respect to the camera / detection unit 10. The left side connecting member 80L similarly has a belt portion 82L and an angle holding member 81L.

[0168] Figure 2B is a view that shows the shape of the belt portion 82L of the left side connecting member 80L and the belt portion 82R of the right side connecting member 80R. In Figure 2B , the angle holding members 81L and 81R are shown as transparent members to show the shape of the belt portions 82L and 82R. The belt portion 82L is provided with a left side connecting surface 83L and a cable 84 that are arranged on the left side of the user's body (the right side in Figure 2B ) when the user wears the camera body 1. The belt portion 82R is provided with a right side connecting surface 83R that is arranged on the right side of the user's body (the left side in Figure 2B ).

[0169] The left-side connecting surface 83L is connected to the corner holding member 81L, and its cross-sectional shape is elliptical rather than a perfect circle. The right-side connecting surface 83R also has a similar elliptical shape. The right-side connecting surface 83R and the left-side connecting surface 83L are symmetrically arranged in an inverted V shape. That is, in the Figure 2B middle, the distance between the right-side connecting surface 83R and the left-side connecting surface 83L becomes shorter from the lower side toward the upper side. Thereby, since the long axis direction of the cross section of the left-side connecting surface 83L and the right-side connecting surface 83R matches the user's body in the case where the user hangs the camera body 1, the strap portions 82L and 82R comfortably contact the user's body, and movement of the imaging / detecting unit 10 in the left-right direction and the front-back direction can be prevented.

[0170] A cable (power supply member) 84 is routed inside the strap portion 82L, and electrically connects the battery unit 90 and the imaging / detecting unit 10. The cable 84 connects the power source of the battery unit 90 to the imaging / detecting unit 10, or transmits an electric signal with an external device.

[0171] Figure 2C is a rear view of the imaging / detecting unit 10. Figure 2C shows the side that contacts the user's body. That is, Figure 2C is a view from the opposite side of Figure 2A . Therefore, the positional relationship between the right-side connecting member 80R and the left-side connecting member 80L is reversed from Figure 2A .

[0172] The imaging / detecting unit 10 is provided on the back side with a power switch 11, an imaging mode switch 12, and chest contact pads 18a and 18b. The power switch 11 is used to switch on / off of the power source of the camera body 1. Although the power switch 11 of the present embodiment is a slide lever type, it is not limited thereto. For example, the power switch 11 can be a push type switch, or can be a switch that is integrally formed with a slide cover (not shown) of the imaging lens 16.

[0173] The imaging mode switch (change member) 12 is used to change the imaging mode, that is, to change the mode related to the imaging operation. In the present embodiment, the imaging mode switch 12 is used to select the imaging mode from a still image mode, a video image mode, and a preset mode set using the display device 800 as described below. In the present embodiment, the imaging mode switch 12 is a switch that can select Figure 2CThe slide lever switch shown is one of "photo", "normal", and "preset". The camera mode is changed to a still image mode by sliding to "photo", to a video image mode by sliding to "normal", and to a preset mode by sliding to "preset". Note that the configuration of the camera mode switch 12 is not limited to the present embodiment, as long as the switch can change the camera mode. For example, the camera mode switch 12 can be composed of three buttons of "photo", "normal", and "preset".

[0174] The chest contact pads (fixing members) 18a and 18b touch the user's body when the camera / detection unit 10 exerts force against the user's body. As shown in Figure 2A The camera / detection unit 10 is formed so that the total length in the lateral (left-right) direction will become longer than the total length in the vertical (up-down) direction when the camera body 1 is worn, as shown. The chest contact pads 18a and 18b are arranged near the right and left ends of the camera / detection unit 10, respectively. This arrangement reduces rotational blur in the left-right direction during the camera operation of the camera body 1. In addition, the chest contact pads 18a and 18b prevent the power switch 11 and the camera mode switch 12 from touching the user's body. Furthermore, the chest contact pads 18a and 18b prevent heat transfer to the user's body even in the case where the camera / detection unit 10 is heated due to long-time camera operation, and serve to adjust the angle of the camera / detection unit 10.

[0175] Figure 2D is a top view showing the camera / detection unit 10. As shown in Figure 2D A face direction detection window 13 is provided in the central portion of the top surface of the camera / detection unit 10, and the chest contact pads 18a and 18b protrude from the camera / detection unit 10.

[0176] Figure 2E is a diagram showing the structure of a face direction detection unit 20 arranged inside the camera / detection unit 10 and below the face direction detection window 13. The face direction detection unit 20 is provided with an infrared LED 22 and an infrared condenser lens 26. The face direction detection unit 20 is also provided with an infrared LED lighting circuit 21 and an infrared detection device 27, which will be described later. Figure 5 The infrared LED lighting circuit 21 and the infrared detection device 27 shown.

[0177] The infrared LED 22 projects infrared light 23 Figure 5 ) toward the user. The infrared condenser lens 26 images reflected light 25 Figure 5 ) from the user, which is reflected when the infrared light 23 is projected from the infrared LED 22, onto a sensor (not shown) of the infrared detection device 27.

[0178] Figure 2F is a diagram showing the state where the user wears the camera body 1, seen from the left side of the user.

[0179] The angle adjustment button 85L is provided in the angle holding member 81L and is used to adjust the angle of the imaging / detecting unit 10. The angle adjustment button (not shown in Figure 2F ) is provided in the angle adjustment button 85L in a symmetrical position within the opposite angle holding member 81R.

[0180] Although the angle adjustment buttons are actually visible in Figure 2A , Figure 2C and Figure 2D , these angle adjustment buttons are omitted to simplify the explanation.

[0181] When the angle holding member 81L is moved upward or downward in Figure 2F while the angle adjustment button 85L is pressed, the user can change the angle between the imaging / detecting unit 10 and the angle holding member 81L. The right side is the same as the left side. Furthermore, the protruding angle of the chest contact pads 18a and 18b can be changed. The functions of these two angle changing members (the angle adjustment button and the chest contact pads) can adjust the imaging / detecting unit 10 so that the optical axis of the imaging lens 16 is horizontally maintained regardless of individual differences in the shape of the chest position.

[0182] Figure 3A , Figure 3B and Figure 3C are diagrams showing details of the battery unit 90. Figure 3A is a partially perspective rear view showing the battery unit 90.

[0183] As shown in Figure 3A , the left battery 94L and the right battery 94R are symmetrically installed within the battery unit 90 to achieve weight balance. In this way, since the left battery 94L and the right battery 94R are symmetrically arranged with respect to the central portion of the battery unit 90, weight balance in the left-right direction is achieved and the position of the camera body 1 is prevented from shifting. It should be noted that the battery unit 90 can be installed with a single battery.

[0184] Figure 3B is a top view showing the battery unit 90. Also in Figure 3B , the batteries 94L and 94R are shown as transparent members.

[0185] As shown in Figure 3B , since the batteries 94L and 94R are symmetrically arranged on both sides of the spine-avoiding cutout 93, the user can wear the relatively heavy battery unit 90 without any burden.

[0186] Figure 3C is a rear view of the battery unit 90. Figure 3C is a diagram seen from the side of the body of the user who touches, i.e., is a diagram seen from Figure 3Athe opposite side of the user. As shown in Figure 3C

[0187] Figure 4 is a functional block diagram showing the camera body 1. Hereinafter, the processing performed by the camera body 1 will be roughly explained using Figure 4 . Details will be explained later. As shown in Figure 4 , the camera body 1 is provided with a face direction detection unit 20, a recording direction / field angle determination unit 30, an imaging unit 40, an image extraction / development unit 50, a main recording unit 60, a transmission unit 70, and a second controller 111. These functional blocks are realized by the control of a general control CPU 101 Figure 5 ) that controls the entire camera body 1.

[0188] The face direction detection unit 20 (observation direction detection unit) is a functional block performed by the above-described infrared LED 22, infrared detection device 27, and the like. The face direction detection unit 20 estimates an observation direction by detecting a face direction, and passes the observation direction to the recording direction / field angle determination unit 30.

[0189] The recording direction / field angle determination unit (recording direction determination unit) 30 determines information about a position and a region to be extracted from an image captured by the imaging unit 40 by performing various calculations based on the observation direction estimated by the face direction detection unit 20. Then, the information is passed to the image extraction / development unit 50.

[0190] The imaging unit 40 forms a wide-angle image of a subject and passes the image to the image extraction / development unit 50. The image extraction / development unit (development unit) 50 extracts an image that the user looks at from the image passed from the imaging unit 40 by using the information passed from the recording direction / field angle determination unit 30. Then, the image extraction / development unit 50 develops the extracted image and passes the developed image to the main recording unit 60.

[0191] The main recording unit 60 is a functional block constituted by the main memory 103 Figure 5 and the like, records image information, and passes the image information to the transmission unit 70 at a required timing. The transmission unit (image output unit) 70 is wirelessly connected to and communicates with predetermined communication parties such as a display device Figure 1D 800, a calibrator 850, and a simplified display device 900.

[0192] ​The display device 800 is connectable to the transmission unit 70 through a high-speed wireless LAN (hereinafter referred to as "high-speed wireless network"). In the present embodiment, the high-speed wireless network adopts wireless communication corresponding to the IEEE 802.11ax (Wi-Fi 6) standard. On the other hand, wireless communication corresponding to other standards such as the Wi-Fi 4 standard and the Wi-Fi 5 standard can be adopted. In addition, the display device 800 can be a dedicated device developed for the camera body 1, or can be a general smart phone, a tablet terminal, or the like.

[0193] In addition, the display device 800 can be connected to the transmission unit 70 through a low-power wireless network, can be connected to the transmission unit 70 through both the high-speed wireless network and the low-power wireless network, or can be connected to the transmission unit 70 while switching the networks. In the present embodiment, a large amount of data such as image files of video images as described later is transmitted through the high-speed wireless network, and a small amount of data and data that does not need to be transmitted quickly are transmitted through the low-power wireless network. In the present embodiment, although Bluetooth is used for the low-power wireless network, other short-distance wireless communication such as NFC (Near Field Communication) can be adopted.

[0194] The calibrator 850 performs initial setting and individual setting of the camera body 1, and is connectable to the transmission unit 70 through the high-speed wireless network in the same manner as the display device 800. Details of the calibrator 850 are described later. In addition, the display device 800 can have the function of the calibrator 850.

[0195] The simplified display 900 is connectable to the transmission unit 70, for example, through only the low-power wireless network. Although the simplified display 900 cannot communicate video images with the transmission unit 70 due to time limitation, the simplified display 900 can transmit the photographing start / stop timing, and can be used for image inspection at the level of composition inspection. In addition, the simplified display 900 can be a dedicated device developed for the camera body 1 like the display device 800, or can be a smart watch.

[0196] Figure 5 is a block diagram showing the hardware structure of the camera body 1. In addition, the structures and functions described in Figures 1A to 1C are denoted by the same reference numerals, and detailed description of these structures and functions will be omitted.

[0197] As shown in Figure 5 , the camera body 1 is provided with a general control CPU 11, a power switch 11, a photographing mode switch 12, a face direction detection window 13, an activation switch 14, a stop switch 15, a photographing lens 16, and an LED 17.

[0198] The camera body 1 is provided with a face direction detection unit 20 ( Figure 4 )'s infrared LED lighting circuit 21, infrared LED 22, infrared focusing lens 26 and infrared detection device 27.

[0199] Furthermore, the camera body 1 is provided with an image pickup unit 40 ( Figure 4 ) (which consists of an imaging driver 41, a solid-state image sensor 42, and an image signal processing circuit 43) and a transmitting unit 70 ( Figure 4 )(which consists of a low-power wireless communication unit 71 and a high-speed wireless communication unit 72).

[0200] In this embodiment, although the camera body 1 has a single imaging unit 40, the camera body 1 may have two or more imaging units to capture 3D images, capture images with a wider field angle than that obtained by a single imaging unit, or capture images in different directions.

[0201] The camera body 1 is provided with various memories such as a large-capacity nonvolatile memory 51, an internal nonvolatile memory 102, and a main memory 103. In addition, the camera body 1 is provided with an audio processor 104, a speaker 105, a vibrator 106, an angular velocity sensor 107, an acceleration sensor 108, and various switches 110.

[0202] use Figure 2C The aforementioned switches such as the power switch 11 are connected to the overall control CPU 101. The overall control CPU 101 controls the entire camera body 1. Figure 4 The recording direction / field angle determination unit 30 , the image extraction / development unit 50 , and the second controller 111 in the image processing apparatus are implemented by the overall control CPU 101 .

[0203] Infrared LED lighting circuit 21 controls infrared LED 22 ( Figure 2E ) is turned on and off to control the projection of infrared light 23 directed from the infrared LED 22 to the user.

[0204] The face direction detection window 13 is formed of a visible light cut filter that cuts off most visible light and sufficiently allows infrared light 23 belonging to the infrared region and its reflected light 25 to pass therethrough. The infrared condensing lens 26 condenses the reflected light 25 .

[0205] The infrared detection device (infrared detection unit) 27 has a sensor that detects the reflected light 25 focused by the infrared focusing lens 26. The sensor converts an image formed by the focused reflected light 25 into sensor data and transmits the sensor data to the overall control CPU 101.

[0206] like Figure 1BAs shown, in a case where the user wears the camera body 1, the face direction detection window 13 is located below the user's chin. Therefore, as shown in FIG. 2B, the face direction detection window 13 is located below the user's chin when the user wears the camera body 1. Figure 5 As shown, infrared light 23 projected from the infrared LED 22 transmits through the face direction detection window 13, and the user's chin vicinity is irradiated with the infrared light 23. In addition, reflected light 25 reflected from the infrared irradiation surface 24 transmits through the face direction detection window 13, and is condensed by the infrared condensing lens 26 onto a sensor in the infrared detection device 27.

[0207] The various switches 110 are not shown in FIG. 1. The various switches 110 are used to perform functions unrelated to the present embodiment. Figures 1A to 1C The various switches 110 are not shown in FIG. 1. The various switches 110 are used to perform functions unrelated to the present embodiment.

[0208] The imaging driver 41 includes a timing generator that generates various timing signals, outputs the timing signals to each section related to the imaging operation, and drives the solid-state image sensor 42. The solid-state image sensor 42 outputs a signal obtained by photoelectrically converting a subject image formed through the imaging lens 16 (FIG. 1) to the image signal processing circuit 43. Figure 1A

[0209] The image signal processing circuit 43 generates imaging data by applying clamp processing, A / D conversion processing, and the like to the signal from the solid-state image sensor 42, and outputs the imaging data to the overall control CPU 101.

[0210] The internal nonvolatile memory 102 is constituted by a flash memory or the like, and stores a boot program used by the overall control CPU 101, and setting values of various program modes. In the present embodiment, setting values of an observation field of view (angle of view) and setting values of an effect level of the image stabilization processing are recorded.

[0211] The main memory 103 is constituted by a RAM or the like, and temporarily stores image data in processing and a calculation result of the overall control computer 101.

[0212] The large-capacity nonvolatile memory 51 stores image data. In the present embodiment, the large-capacity nonvolatile memory 51 is a non-removable semiconductor memory. However, the large-capacity nonvolatile memory can be constituted by a removable storage medium such as an SD card, and can be used together with the internal nonvolatile memory 102.

[0213] The low-power wireless communication unit 71 exchanges data with the display device 800, the calibrator 850, and the simplified display apparatus 900 through a low-power wireless network. The high-speed wireless communication unit 72 exchanges data with the display device 800 and the calibrator 850 through a high-speed wireless network.

[0214] The audio processor 104 processes external sounds (analog signals) collected by the microphones 19L and 19R and generates an audio signal.​

[0215] In order to notify the user of the state of the camera body 1 and warn the user, the LED 17 emits light, the speaker 105 outputs sound, and the vibrator 106 vibrates.

[0216] The angular velocity sensor 107 uses a gyroscope or the like and detects the movement of the camera body 1 as gyroscope data. The acceleration sensor 108 detects the posture of the imaging / detecting unit 10.

[0217] Figure 6 is a block diagram showing the hardware structure of the display device 800. The components described above are denoted by the same reference numerals, and the explanation of these components will be omitted to simplify the explanation. As shown in Figure 1D the display device 800 is provided with a display device controller 801, an A button 802, a display unit 803, a B button 804, a face sensor 806, an angular velocity sensor 807, an acceleration sensor 808, an image signal processing circuit 809, and various switches 811. Figure 6

[0218] Further, the display device 800 is provided with an internal nonvolatile memory 812, a main memory 813, a large-capacity nonvolatile memory 814, a speaker 815, a vibrator 816, an LED 817, an audio processor 820, a low-power wireless communication unit 871, and a high-speed wireless communication unit 872. The above components are connected to the display device controller 801.

[0219] The display device controller 801 is constituted by a CPU and controls the display device 800.

[0220] The image signal processing circuit 809 assumes the same functions as the imaging driver 41, the solid-state image sensor 42, and the image signal processing circuit 43, etc. inside the camera body 1. The image signal processing circuit 809 constitutes the built-in camera 805 together with the built-in camera lens 805a. Figure 1D The display device controller 801 processes the data output from the image signal processing circuit 809. The content of the processing of the data will be explained later.

[0221] The various switches 811 are used to perform functions irrelevant to the present embodiment.

[0222] The angular velocity sensor 807 uses a gyroscope or the like and detects the movement of the display device 800. The acceleration sensor 808 detects the posture of the display device 800.

[0223] The internal nonvolatile memory 812 is constituted by a flash memory or the like and stores a boot program used by the display device controller 801, and setting values of various program modes.

[0224] ​The main memory 813 is constituted by a RAM or the like, and temporarily stores image data in processing and a result of calculation of the image signal processing circuit 809. In the present embodiment, gyro data detected by the angular velocity sensor 107 at the time of shooting of each frame is stored in the main memory 813 in association with the frame while a video image is being recorded.

[0225] The large-capacity nonvolatile memory 51 stores image data of the display device 800. In the present embodiment, the large-capacity nonvolatile memory 814 is constituted by a detachable memory such as an SD card. It should be noted that, like the large-capacity nonvolatile memory 51 in the camera body 1, the large-capacity nonvolatile memory 814 can be constituted by a fixed memory.

[0226] In order to notify the user of the state of the display device 800 and to warn the user, the LED 817 emits light, the speaker 815 outputs sound, and the vibrator 816 vibrates.

[0227] The audio processor 820 processes external sound (analog signal) collected by the microphones 819L and 819R and generates an audio signal.

[0228] The low-power wireless communication unit 871 exchanges data with the camera body 1 through a low-power wireless network. The high-speed wireless communication unit 872 exchanges data with the camera body 1 through a high-speed wireless network.

[0229] The face sensor (face detection unit) 806 is provided with an infrared LED lighting circuit 821 and an infrared LED 822, an infrared condenser lens 826, and an infrared detection device 827.

[0230] The infrared LED lighting circuit 821 has a function similar to that of the infrared LED lighting circuit 21 in the camera body 1, and controls lighting and extinguishing of the infrared LED 822 to control projection of infrared light 823 directed from the infrared LED 822 toward the user. The infrared condenser lens 826 condenses reflected light 825. Figure 5

[0231] The infrared detection device (infrared detection unit) 827 has a sensor that detects the reflected light 825 condensed by the infrared condenser lens 826. The sensor converts the reflected light 825 after condensation into sensor data, and transmits the sensor data to the display device controller 801.

[0232] In the case where the face sensor 806 shown in FIG. 8 is directed toward the user, as shown in FIG. 9, the infrared LED 822 is lit, and the infrared light 823 is projected toward the user. The reflected light 825 is condensed by the infrared condenser lens 826, and the sensor of the infrared detection device 827 detects the reflected light 825. Figure 1D Figure 6 ​​As shown, an infrared irradiation surface 824, which is the entire face of the user, is irradiated with infrared light 823 projected from the infrared LED 822. Furthermore, reflected light 825 reflected from the infrared irradiation surface 824 is condensed by the infrared condenser lens 826 onto a sensor in the infrared detection device 827.

[0233] Other functions 830 are functions of the smart phone such as a telephone function, which are not related to the present embodiment.

[0234] How the camera body 1 and the display device 800 are used will be described below. Figure 7A is a flowchart schematically showing the imaging / recording processing according to the first embodiment performed by the camera body 1 and the display device 800.

[0235] To assist the description, in Figure 7A the reference numerals of the units that perform the processing in each step are shown on the right side of each step in Figure 4 or Figure 5 . That is, the steps S100 to S700 in Figure 7A are performed by the camera body 1, and the steps S800 to S1000 in Figure 7A are performed by the display device 800.

[0236] When the power switch 11 is set to ON and the power of the camera body 1 becomes ON, the overall control CPU 101 is activated and reads a boot program from the internal nonvolatile memory 102. Thereafter, in step S100, the overall control CPU 101 performs a preparation processing of making settings of the camera body 1 before the imaging operation. Details of the preparation processing will be described later using Figure 7B .

[0237] In step S200, a face direction detection processing of estimating the viewing direction based on the face direction detected by the face direction detection unit 20 is performed. Details of the face direction detection processing will be described later using Figure 7C . This processing is performed at a predetermined frame rate.

[0238] In step S300, the recording direction / field angle determination unit 30 performs a recording direction / region determination processing. Details of the recording direction / region determination processing will be described later using Figure 7D . In step S400, the imaging unit 40 captures an image and generates imaging data.

[0239] In step S500, the image extraction / visualization unit 50 extracts an image from the imaging data generated in step S400 according to the recording direction / field angle information determined in step S300, and performs a recording region visualization processing of visualizing the extracted region. Details of the recording region visualization processing will be described later using Figure 7EThe details of the recording area development process will be described.

[0240] In step S600, the main recording unit (image recording unit) 60 performs a main recording process of storing the image developed in step S500 as image data in the main memory 103. Figure 14 To explain the details of master record processing.

[0241] In step S700, the transmission unit 70 performs a transmission process to the display device 800, which wirelessly transmits the image recorded in step S600 to the display device 800 at a specified timing. Figure 16 The details of the transmission process to the display device 800 will be described.

[0242] The steps from step S800 are executed by the display device 800. In step S800, the display device controller 801 performs an optical correction process for correcting the optical aberration of the image transmitted from the camera body 1 in step S700. Figure 17 The details of the optical correction processing will be described.

[0243] In step S900, the display device controller 801 applies image stabilization processing to the image for which the optical aberration has been corrected in step S800. Figure 19 The details of the image stabilization processing are explained below.

[0244] It should be noted that the order of step S800 and step S900 may be reversed. That is, the image stabilization process may be performed before the optical correction process.

[0245] In step S1000, the display device controller (video recording unit) 801 performs auxiliary recording processing that records the image to which the optical correction processing in step S800 and the image stabilization processing in step S900 are applied into the large-capacity nonvolatile memory 814. Then, the display device controller 801 ends the processing.

[0246] Next, we will use Figures 7B to 7F and other figures are described in detail in the order of processing Figure 7A The processing (subroutine) in each step. Figure 7B It shows Figure 7A Flowchart of the subroutine for preparing the process in step S100. Figures 2A to 2F and Figure 5 The components shown are used to illustrate the process.

[0247] It is determined in step S101 whether the power switch 11 is on. In the case where the power is off, the process waits. In the case where the power becomes on, the process proceeds to step S102. In step S102, it is determined which mode is selected by the imaging mode switch 12. As a result of the determination, in the case where the mode selected by the imaging mode switch 12 is the video image mode, the process proceeds to step S103.

[0248] In step S103, various setting values of the video image mode are read from the internal nonvolatile memory 102, and the setting values are stored in the main memory 103. Then, the process proceeds to step S104. The various setting values of the video image mode include the angle of field setting value V ang and the image stabilization level. In the present embodiment, the angle of field setting value V ang is set to 90° in advance. The image stabilization level is selected from "strong", "medium", and "off".

[0249] In step S104, the operation of the imaging driver 41 used for the video image mode is started. Then, the process exits the subroutine. As a result of the determination in step S102, in the case where the mode selected by the imaging mode switch 12 is the still image mode, the process proceeds to step S106.

[0250] In step S106, various setting values of the still image mode are read from the internal nonvolatile memory 102, and the setting values are stored in the main memory 103. Then, the process proceeds to step S107. The various setting values of the still image mode include the angle of field setting value V ang and the image stabilization level. In the present embodiment, the angle of field setting value V ang is set to 45° in advance. The image stabilization level is selected from "strong", "medium", and "off".

[0251] In step S107, the operation of the imaging driver 41 used for the still image mode is started. Then, the process exits the subroutine.

[0252] As a result of the determination in step S102, in the case where the mode selected by the imaging mode switch 12 is the preset mode, the process proceeds to step S108. The preset mode is one of the three imaging modes which can be changed by the imaging mode switch 12. In the preset mode, the imaging mode of the camera body 1 can be changed by an external device such as the display device 800. That is, the preset mode is used for customizing the imaging operation. Since the camera body 1 is a compact wearable device, there is no operation switch, setting screen, or the like for changing the advanced setting value mounted on the camera body 1. The advanced setting value is changed by the external device such as the display device 800.

[0253] For example, consider a case where a user wants to change the field angle from 90° to 110° while continuously capturing video. In this case, the following operations are required. Since the field angle is set to 90° in normal video mode, the user performs a video capture operation in normal video mode, and once the video capture operation is completed, displays a settings screen on display device 800, and changes the field angle to 110° on this settings screen. However, operating display device 800 during an event is cumbersome.

[0254] On the other hand, in the case where the preset mode is preset to a video image capturing operation at a field angle of 110°, the user can immediately change the field angle in the video image capturing operation to 110° by simply sliding the camera mode switch 12 to “Preset” after finishing the video image capturing operation at a field angle of 90°. That is, the user does not need to pause the current operation and perform the troublesome operation described above.

[0255] It should be noted that the contents of the preset mode may include, in addition to the field angle, an image stabilization level ("strong", "medium", or "off") and a setting value for voice recognition which is not described in this embodiment.

[0256] In step S108, various setting values ​​of the preset mode are read from the internal nonvolatile memory 102 and stored in the main memory 103. Then, the process proceeds to step S109. Various setting values ​​of the preset mode include the field angle setting value V ang , and the level of image stabilization selected from Strong, Medium, and Off.

[0257] In step S109, the operation of the image pickup driver 41 for the preset mode is started. Then, the process exits this subroutine.

[0258] Afterwards, you will use Figure 13 Various setting values ​​of the video image mode read in step S103 will be described. Figure 13 1 is a diagram showing a menu screen for setting various setting values ​​of the video image mode displayed on the display unit 803 of the display device 800 before the image capturing operation of the camera body 1. Figure 1D The components are denoted by the same reference numerals, and description of these components will be omitted. The display unit 803 has a touch panel function, and description will be made assuming that the display unit 803 functions by a touch operation such as a slide operation.

[0259] like Figure 13The menu screen includes a preview screen 831, a zoom lever 832, a record start / stop button 833, a switch 834, a battery remaining amount indicator 835, a button 836, a lever 837, and an icon display area 838. The user can check the image captured by the camera body 1 on the preview screen 831, the zoom amount, and the angle of field.

[0260] The user can change the zoom setting (angle of field) by moving the zoom lever 832 to the right or to the left. The present embodiment illustrates the case where the angle of field setting value V ang can be set to one of four values, 45°, 90°, 110°, and 130°. On the other hand, the angle of field setting value V ang can be set to a value other than these four values by operating the zoom lever 832.

[0261] The record start / stop button 833 is a toggle switch having both the function of the start switch 14 and the function of the stop switch 15. The switch 834 is used to switch between "off" and "on" of the image stabilization processing. The battery remaining amount indicator 835 displays the remaining amount of the battery of the camera body 1. The button 836 is used to change the mode.

[0262] The lever 837 is used to set the image stabilization level. In the present embodiment, although the image stabilization level can be set to "strong" or "medium", another image stabilization level (e.g., "weak") can be set. Further, the image stabilization level can be set in a stepless manner. A plurality of thumbnail icons used for preview are displayed in the icon display area 838.

[0263] Figure 7C is a flowchart showing a subroutine of the face direction detection processing in step S200 of Figure 7A . Before explaining the details of the processing, the face direction detection method using infrared light will be explained using Figures 8A to 8K .

[0264] Figure 8A is a diagram showing a visible light image of the face of the user looking at the position of the face direction detection window 13. Figure 8A The image of Figure 8A includes the front part of the neck 201 above the collarbone, the root of the jaw 202, the chin 203, and the face 204 including the nose.

[0265] Figure 8B is a diagram showing the case where a fluorescent lamp in the room appears as a background in the visible light image of the user shown in Figure 8A . In Figure 8BIn the visible light image, the fluorescent lamp 205 around the user appears. Thus, since various backgrounds appear in the image of the user depending on the use condition, it becomes difficult for the face direction detection unit 20 or the overall control CPU 101 to cut out the face image from the visible light image. On the other hand, although there is a technique of cutting out such an image by using AI or the like, since the overall control CPU 101 needs to have high performance, this technique is not suitable as the camera body 1 of a portable device.

[0266] Thus, the camera body 1 of the first embodiment detects the face of the user using the infrared image. Since the face direction detection window 13 is constituted by a visible light cut filter, visible light is not transmitted therethrough in most cases. Thus, the image obtained by the infrared detection device 27 is different from Figure 8A and Figure 8B .

[0267] Figure 8C is a view showing an infrared image obtained by imaging the user shown in Figure 8B and the fluorescent lamp as a background through the face direction detection window 13 onto the sensor of the infrared detection device 27 in a state where the infrared LED 22 is not lit.

[0268] In the infrared image of Figure 8C , the neck and the jaw of the user are dark. On the other hand, since the fluorescent lamp 205 emits an infrared component in addition to visible light, the fluorescent lamp 205 is slightly bright.

[0269] Figure 8D is a view showing an image obtained by imaging the user shown in Figure 8B and the fluorescent lamp as a background through the face direction detection window 13 onto the sensor of the infrared detection device 27 in a state where the infrared LED 22 is lit.

[0270] In the image of Figure 8D , the neck and the jaw of the user are bright. On the other hand, unlike Figure 8C , the brightness around the fluorescent lamp 205 is not changed. Figure 8E is a view showing a difference image calculated by subtracting the image of Figure 8D from the image of Figure 8C . The face of the user appears.

[0271] Thus, the overall control CPU (image obtaining unit) 101 obtains a difference image (hereinafter referred to as a face image) by calculating the difference between the image formed on the sensor of the infrared detection device 27 in a state where the infrared LED 22 is lit and the image formed on the sensor in a state where the infrared LED 22 is not lit.

[0272] The face direction detection unit 20 of this embodiment adopts a method of obtaining a face image by extracting infrared reflection intensity as a two-dimensional image using the infrared detection device 27. The sensor of the infrared detection device 27 adopts a similar structure to a general image sensor, and a face image is obtained frame by frame. A vertical synchronization signal (hereinafter referred to as a V signal) that obtains frame synchronization is generated by the infrared detection device 27, and is output to the overall control CPU 101.

[0273] Figure 9 is a timing chart showing the lighting and extinguishing of the infrared LED 22 and the related signals. In Figure 9 the V signal output from the infrared detection device 27, the H position of the image signal output from the sensor of the infrared detection device 27, the IR-ON signal output from the overall control CPU 101 to the infrared LED lighting circuit 21, and the imaging data output from the sensor of the infrared detection device 27 to the overall control CPU 21 are shown in order from top to bottom. The time axis of the horizontal axis of these four signals is the same.

[0274] The timing of obtaining frame synchronization and the timing of lighting and extinguishing of the infrared LED 22 when the V signal becomes high. Figure 9 The first face image obtaining period tl and the second face image obtaining period t2 are shown.

[0275] The infrared detection device 27 controls the operation of the sensor so that, as shown in Figure 9 , the H position of the image signal will be synchronized with the V signal. Since the sensor of the infrared detection device 27 adopts a similar structure to a general image sensor and its operation is well known as described above, a detailed explanation of the control method is omitted.

[0276] The overall control CPU 101 controls the switching of the IR-ON signal between high and low in synchronization with the V signal. Specifically, the overall control CPU 101 outputs the IR-ON signal of low to the infrared LED lighting circuit 21 during the period tl, and outputs the IR-ON signal of high to the infrared LED lighting circuit 21 during the second period t2.

[0277] During the high period of the IR-ON signal, the infrared LED lighting circuit 21 lights the infrared LED 22 to project infrared light 23 to the user. On the other hand, during the low period of the IR-ON signal, the infrared LED lighting circuit 21 extinguishes the infrared LED 22.

[0278] The vertical axis of the imaging data represents the signal intensity of the light receiving amount as reflected light 25. Since the infrared LED 22 is not lit during the first period tl, there is no reflected light from the face of the user and a face image is obtained as Figure 8COn the other hand, since the infrared LED 22 is turned on during the second time period t2, reflected light 25 is generated from the user's face, and the image data shown in FIG. Figure 8D Therefore, the signal intensity in the time period t2 is greater than the signal intensity in the time period t1 by the reflected light 25 from the user's face.

[0279] By subtracting the image data during the first time period t1 from the image data during the second time period t2, we obtain Figure 9 As a result of the subtraction, facial image data is obtained in which only the component of the reflected light 25 from the user's face is extracted.

[0280] Figure 7C Shown including the use Figures 8C to 8E and Figure 9 The face direction detection process in step S200 of the operation is described.

[0281] In step S201, when the V signal output from the infrared detection device 27 goes high, a timing V1 is obtained at which the first time period t1 begins. Upon obtaining the timing V1, the process proceeds to step S202. In step S202, the IR-ON signal is set to low and output to the infrared LED lighting circuit 21. Consequently, the infrared LED 22 is not lit.

[0282] In step S203, one frame of image pickup data output during the first time period t1 is read from the infrared detection device 27. This image data is temporarily stored in the main memory 103 as frame 1 (Frame 1).

[0283] In step S204, when the V signal output from the infrared detection device 27 becomes high, the timing V2 at which the second period t2 starts is obtained. When the timing V2 is obtained, the process proceeds to step S205.

[0284] In step S205, the IR-ON signal is set high and output to the infrared LED lighting circuit 21. Thereby, the infrared LED 22 lights up.

[0285] In step S206, one frame of image pickup data outputted from the infrared detection device 27 during the second period t2 is read. This image data is temporarily stored in the main memory 103 as frame 2.

[0286] In step S207, the IR-ON signal is set to low and output to the infrared LED lighting circuit 21. Thereby, the infrared LED 22 is not lit.

[0287] In step S208, frame 1 and frame 2 are read from the main memory 103, and the value from the frame 1 and the frame 2 is calculated by subtracting frame 1 from frame 2. Figure 9 The light intensity Fn of the reflected light 25 of the user corresponding to the illustrated face image. This processing is generally called black subtraction.

[0288] In step S209, the throat position (head rotation center) is extracted from the light intensity Fn. First, the overall control CPU (segmentation unit) 101 segments the face image into the regions to be used for the Figure 8F illustrated multiple distance regions.

[0289] Figure 8F is a graph showing the result obtained by adjusting the Figure 8E illustrated difference image to the scale of the light intensity of the reflected light 25 of the infrared light 23 projected to the face and neck of the user. Figure 8F illustrates the light intensity distribution with respect to each part of the face and neck of the user.

[0290] Figure 8F The face image on the left side of Figure 8E illustrates the light intensity distribution of the reflected light 25 in the illustrated face image. The Xf axis is added in the direction from the central part of the neck of the user toward the chin. In the Figure 8F graph on the right side of Figure 8F , the horizontal axis illustrates the light intensity on the Xf axis of the face image, and the vertical axis illustrates the Xf axis. The light intensity illustrated by the horizontal axis increases as it moves to the right. According to the light intensity, the face image of

[0291] Region 211 is a region in which the light intensity is the strongest, and is illustrated by white in the gray step. Region 212 is a region in which the light intensity is slightly decreased compared to region 211, and is illustrated by very bright gray in the gray step. Region 213 is a region in which the light intensity is further decreased compared to region 212, and is illustrated by bright gray in the gray step.

[0292] Region 214 is a region in which the light intensity is further decreased compared to region 213, and is illustrated by medium gray in the gray step. Region 215 is a region in which the light intensity is further decreased compared to region 214, and is illustrated by slightly dark gray in the gray step.

[0293] Region 216 is a region in which the light intensity is the weakest, and is illustrated by the darkest gray in the gray step. The region above region 216 is illustrated by black that does not exhibit the light intensity.

[0294] The light intensity will be described in detail using 10A to 10D . 10A to 10D is a graph illustrating the movement of the face of the user in the vertical direction, and illustrates the state observed from the left side of the user.

[0295] Figure 10A : This figure shows a state where the user is facing forward. The imaging / detection unit 10 is located in front of the user's clavicle. Furthermore, infrared light 23 from an infrared LED 22 is irradiated onto the lower portion of the user's head from a facial direction detection window 13 installed on the upper portion of the imaging / detection unit 10. The distance Dn from the facial direction detection window 13 to the user's throat 200 above the clavicle, the distance Db from the facial direction detection window 13 to the base of the jaw 202, and the distance Dc from the facial direction detection window 13 to the chin 203 satisfy Dn. <Db<Dc的关系。由于光强度与距离的平方成反比,因此在由反射光25在红外照射面24的传感器上形成的图像中的光强度按喉部200、颌的根部202、以及下巴203的顺序逐渐变弱。此外,由于从面部方向检测窗13到包括鼻子的面部204的距离仍然长于距离Dc,因此与面部204相对应的图像中的光强度变得更弱。也就是说,在 Figure 10A In the case shown, the obtained Figure 8F An image of the light intensity distribution is shown.

[0296] It should be noted that the configuration of the facial direction detection unit 20 is not limited to the configuration shown in the present embodiment, as long as the facial direction of the user can be detected. For example, an infrared pattern can be projected from an infrared LED (infrared pattern irradiation unit) 22, and the sensor of the infrared detection device 27 (infrared pattern detection unit) can detect the infrared pattern reflected from the irradiation target. In this case, it is preferred that the sensor of the infrared detection device 27 is composed of a structured optical sensor. In addition, the sensor of the infrared detection device 27 can be a sensor (infrared phase comparison unit) that compares the phase of the infrared light 23 and the phase of the reflected light 25. For example, a ToF sensor can be used.

[0297] Next, we will use Figure 8G To illustrate Figure 7C Extraction of the throat position in step S209. Figure 8F Superposition representation Figure 10A The figure marks of the parts of the user's body shown, the double circle showing the throat position and the black circle showing the chin position are obtained. Figure 8G Left image.

[0298] The white area 211 and the throat 200 ( Figure 10A ) corresponds to the very bright grey area 212 and the front of the neck 201 ( Figure 10A ) and the light grey area 213 corresponds to the root 202 ( Figure 10A ) corresponds to the middle gray area 214 and the chin 203 ( Figure 10A) and the slightly darker gray area 215 corresponds to the area located on the face 204 ( Figure 10A ) corresponds to the lips and the lower part of the face around the lips in the lower part of the face 204 ( ). In addition, the darkest gray area 216 corresponds to the lower part of the face 204 ( Figure 10A )The nose in the center corresponds to the upper part of the face around the nose.

[0299] Because Figure 10A As shown, the difference between distances Db and Dc is relatively small compared to differences between other distances from the face direction detection window 13 to other parts of the user, so the difference between the reflected light intensities in the light gray area 213 and the medium gray area 214 is also small.

[0300] On the other hand, due to Figure 10A As shown, among the distances from the face direction detection window 13 to the various parts of the user, the distance Dn is the shortest, and therefore the intensity of the reflected light in the white area 211 corresponding to the throat 200 becomes the strongest.

[0301] Therefore, the overall control CPU (setting unit) 101 determines that the area 211 corresponds to the throat 200 and its periphery, and then sets the position 206 (at the center in the lateral direction and closest to the imaging / detection unit 10) to Figure 8G The processing up to this point is performed at Figure 7C The contents are performed in step S209.

[0302] Next, we will use Figure 8G To illustrate Figure 7C Extraction of the chin position in step S210. Figure 8G In the image of FIG. 2 , a medium gray area 214 that is brighter than an area 215 corresponding to a lower portion of the face including the lips of the face 204 includes the chin. Figure 8G The graph on the right side of shows that the light intensity in the region 215 adjacent to the region 214 drops sharply due to the increasing rate of change of the distance from the face direction detection window 13. The overall control CPU 101 determines that the brighter region 214 adjacent to the region 215 where the light intensity drops sharply is the chin region. In addition, the overall control CPU 101 calculates (extracts) the position (defined by the position φ) that is located at the center in the lateral direction of the region 214 and is farthest from the throat position 206. Figure 8G The black circle shown) is used as the chin position 207.

[0303] For example, Figure 8H and Figure 8I Shows the changes when the face is turned to the right. Figure 8H is a diagram showing how to make the user's face face right by Figure 8EThe difference image is calculated in a similar way. Figure 8I is a graph showing the following results, which are obtained by adjusting Figure 8H The shades of the difference image are scaled to suit the light intensity of the reflected component of the infrared light projected onto the user's face and neck, and are obtained by superimposing a double circle showing the throat position 206 as the position of the center of rotation of the head and a black circle showing the chin position 207r.

[0304] Since the user's face is facing right, the area 214 is located on the left side when looking up from the imaging / detection unit 10. Figure 8I The region 214r shown has moved. The region 215 corresponding to the lower portion of the face including the lips of the face 204 has also moved to a region 215r located on the left side when viewed from above from the imaging / detection unit 10.

[0305] Therefore, the overall control CPU 101 determines that the brighter area 214r adjacent to the area 215r where the light intensity drops sharply is the chin area. In addition, the overall control CPU 101 calculates (extracts) the position (given by the position) that is located at the center in the lateral direction of the area 214r and farthest from the throat position 206. Figure 8I The black circle shown) is used as the chin position 207r.

[0306] Then, the control unit 101 obtains the movement angle θr, which represents the movement angle from the throat position 206 to the Figure 8G From the chin position 207 in the image, go right until Figure 8I The rotation movement to the chin position 207r. Figure 8I As shown, the movement angle θr is the angle of movement of the user's face in the lateral direction.

[0307] In step S210 , the angle of the user's face in the lateral direction (hereinafter referred to as the face angle) is calculated based on the chin position detected by the infrared detection device 27 of the face direction detection unit (three-dimensional detection sensor) 20 .

[0308] Next, the detection of an upward-facing face will be described. Figure 10B The figure shows a state in which the user faces the horizontal direction. Figure 10C The figure shows a state in which the user's face is turned upward by 33 degrees from the horizontal direction.

[0309] exist Figure 10B The distance from the face direction detection window 13 to the chin 203 is Ffh, and Figure 10C The distance from the face direction detection window 13 to the chin 203u is Ffu. Since the chin 203u moves upward together with the face, Figure 10C As shown, the distance Ffu becomes longer than the distance Ffh.

[0310] Figure 8J is a diagram showing an image of the user with the face 33° upward from the horizontal direction as seen from the face direction detection window 13. Since the user has the face upward as shown in Figure 10C , the face 204 including the lips and the nose is not seen from the face direction detection window 13 located below the user's chin. The chin 203 and its neck side are seen. Figure 8K shows the distribution of the light intensity of the reflected light 25 when the user is irradiated with the infrared light 23 in the state shown in Figure 10C . Figure 8K The image on the left side of Figure 8E is a diagram showing the result obtained by adjusting the gradation of the difference image calculated by the same method as Figure 8K to fit the scale of the light intensity of the reflected component of the infrared light projected to the face and the neck of the user, and by superimposing the double circle showing the throat position 206 and the black circle showing the chin position 207u. Figure 8F The two graphs in Figure 8G show the change in the concentration of the left image. The left graph is equivalent to the graph in , and the right graph is equivalent to the graph in

[0311] . Figure 8F The six regions 211u, 212u, 213u, 214u, 215u, and 216u in Figure 8K are indicated by adding "u" to the reference numerals of the same light intensity regions shown in Figure 8F . Although the light intensity of the user's chin 203 is included in the medium gray region 214 in Figure 8K , this light intensity moves to the black side and is included in the slightly darker gray region 215u in Figure 10C . In this way, since the distance Ffu is longer than the distance Ffh as shown in , the infrared detection device 27 can detect the light intensity of the reflected light 25 from the chin 203 to weaken in inverse proportion to the square of the distance.

[0312] Next, the detection of the downward face will be described. Figure 10D is a diagram showing the state in which the user has the face 22° downward from the horizontal direction. In Figure 10D , the distance from the face direction detection window 13 to the chin 203d is Ffd.

[0313] Since the chin 203u moves downward together with the face, the distance Ffd becomes shorter than the distance Ffh as shown in Figure 10D , and the light intensity of the reflected light 25 at the chin 203 becomes stronger.

[0314] Returning to Figure 7CIn step S211, the overall control CPU (distance calculation unit) 101 calculates the distance from the chin position to the face direction detection window 13 based on the light intensity of the chin position detected by the infrared detection device 27 of the face direction detection unit (three-dimensional detection sensor) 20. Based on the light intensity, the face angle in the vertical direction is also calculated.

[0315] In step S212, the overall control CPU 101 stores the face angle θh in the lateral direction (first detection direction) obtained in step S210 and the face angle θv in the vertical direction (second detection direction) obtained in step S211 as the three-dimensional viewing direction vi ("i" is an arbitrary reference numeral) of the user into the main memory 103. For example, in the case where the user is viewing the front center, the face angle θh in the lateral direction is 0° and the face angle θv in the vertical direction is 0°. Therefore, the viewing direction vo in this case is represented by the vector information (0°, 0°). Further, in the case where the user is viewing in the right 45-degree direction, the viewing direction vr is represented by the vector information (45°, 0°).

[0316] Although the face angle in the vertical direction is calculated by detecting the distance with respect to the face direction detection window 13 in step S211, the face angle can be calculated by another method. For example, the change in the face angle can be calculated by comparing the levels of the change in the light intensity of the chin 203. That is, the change in the face angle can be calculated by comparing the gradient CDh of the reflected light intensity from the root 202 of the jaw to the chin 203 in the graph of Figure 8G with the gradient CDu of the reflected light intensity from the root 202 of the jaw to the chin 203 in the graph of Figure 8K The change in the face angle is calculated by comparing the gradient CDh of the reflected light intensity from the root 202 of the jaw to the chin 203 in the graph of

[0317] Figure 7D is a graph showing the target field of view 125 set in the super wide-angle image photographed by the imaging unit 40 in step S300 in Figure 7A will first be described using Figure 11A to explain the super wide-angle image that is the object of determining the recording direction and the recording area in the present embodiment.

[0318] In the camera body 1 of the present embodiment, the imaging unit 40 photographs a super wide-angle image of the periphery of the imaging / detection unit 10 using the super wide-angle imaging lens 16. An image of the viewing direction can be obtained by extracting a part of this super wide-angle image.

[0319] Figure 11A is a graph showing the target field of view 125 set in the super wide-angle image photographed by the imaging unit 40 in the case where the user faces the front. As Figure 11AAs illustrated, the pixel region 121 that can be shot by the solid-state image sensor 42 is a rectangular region. Further, the effective projection region (predetermined region) 122 is a region of a semispherical image that is circular as an image projected on the solid-state image sensor 42 by the imaging lens 16. The imaging lens 16 is adjusted so that the center of the pixel region 121 will coincide with the center of the effective projection region 122.

[0320] The outermost periphery of the circular effective projection region 122 shows a position of a visual field angle of 180°. In a case where the user is looking at the center of both the vertical direction and the horizontal direction, the angle range of the target visual field 125 that is shot and recorded becomes 90° (half of the visual field angle) with the center of the effective projection region 122 as the center. It should be noted that the imaging lens 16 of the present embodiment can also introduce light outside the effective projection region 122, and can project light within a maximum visual field angle (about 192°) to the solid-state image sensor 42 to form a fisheye image. However, the optical performance greatly decreases in a region outside the effective projection region 122. For example, the resolution sharply decreases, the light amount decreases, and the distortion increases. Therefore, in the present embodiment, an image of the observation direction is extracted as a recording region from only the inside of the image projected on the pixel region 121 from within the semispherical image displayed on the effective projection region 122 (hereinafter referred to simply as a super wide-angle image).

[0321] Since the size of the effective projection region 122 in the vertical direction is larger than the size of the short side of the pixel region 121, in the present embodiment, the upper end and the lower end of the image in the effective projection region 122 are outside the pixel region 121. However, the relationship between the respective regions is not limited thereto. For example, the optical system can be designed so that the entire effective projection region 122 will be included in the pixel region 121 by changing the configuration of the imaging lens 16.

[0322] The ineffective pixel region 123 is a portion in the pixel region 121 that is not included in the effective projection region 122. The target visual field 125 shows a region of an image of the observation direction of the user that will be extracted from the super wide-angle image. In Figure 11A In the example illustrated, the target visual field 125 is prescribed by respective left and right upper and lower field angles of 45° (visual field angle of 90°) with the observation direction as the center. In Figure 11A In the example of Fig. 10, since the user is facing the front, the center of the target visual field 125 (observation direction Vo) coincides with the center of the effective projection region 122.

[0323] Figure 11A The super wide-angle image illustrated includes an A subject 131 that is a child, a B subject 132 showing a step that the child (A subject) is attempting to climb, and a C subject 133 that is a locomotive-type playground equipment.

[0324] Next, we will explain Figure 7A Details of the recording direction / area determination processing in step S300 are shown. Figure 7D Shown in order to use Figure 11A The recording direction / area determination process is performed by extracting the image of the observation direction from the ultra-wide-angle image. Figures 12A to 12G This process will be described.

[0325] In step S301, a preset field angle setting value V is obtained by reading from the main memory 103. ang In this embodiment, the internal non-volatile memory 102 stores all available field angles (45°, 90°, 110°, and 130°) as field angle setting values ​​V ang The image extraction / development unit 50 extracts the image data set by the field angle setting value V from the ultra-wide-angle image. ang In addition, an image of the viewing direction in the defined area is established. Figure 7B The field angle setting value V included in the various setting values ​​read from the internal nonvolatile memory 102 in one of steps S103, S106, and S108 ang , and stores it in the main memory 103.

[0326] Furthermore, in step S301, the observation direction vi determined in step S212 is determined as the recording direction, an image in the target field of view 125 whose center is specified by the observation direction vi and whose area is specified by the obtained field angle setting value V is extracted from the ultra-wide-angle image, and the extracted image is stored in the main memory 103. ang To limit.

[0327] For example, in the field angle setting value V ang is 90°, and through the face direction detection process ( Figure 7C ) detects the observation direction vo (vector information (0°, 0°)), a target field of view 125 (the center of which coincides with the center O of the effective projection area 122 and has an angular width of 90° in the horizontal and vertical directions) is established. Figure 11A ). Figure 11B It is shown from Figure 11A That is, the overall control CPU (relative position setting unit) 101 sets the angle of the facial direction detected by the facial direction detection unit 20 as the observation direction vi, which is vector information indicating the relative position of the target field of view 125 with respect to the ultra-wide-angle image.

[0328] In the case of the observation direction vo, since the influence of the optical distortion caused by the camera lens 16 is mostly ignored, the shape of the target field of view 125 established is the same as the target field of view 125o ( after the distortion is converted in step S303 described later) Figure 12A ) are almost the same in shape. Hereinafter, the target field of view after applying the distortion transformation in the case of the viewing direction vi is referred to as the target field of view 125i.

[0329] In step S302, the preset image stabilization level is obtained by reading from the main memory 103. In this embodiment, the image stabilization level is established. Figure 7B The image stabilization level included in the various setting values ​​is read from the internal nonvolatile memory 102 in one of steps S103 , S106 , and S108 , and is stored in the main memory 103 .

[0330] Furthermore, in step S302, the image stabilization margin pixel number P is set based on the obtained image stabilization level. is In the image stabilization process, an image that follows in the direction opposite to the blurring direction is obtained according to the blurring amount of the imaging / detection unit 10. Therefore, in this embodiment, an image stabilization margin required for image stabilization is established around the target field of view 125i.

[0331] In addition, in this embodiment, the internal non-volatile memory 102 stores the image stabilization margin pixel number P. is For example, when the image stabilization level is "medium", the image stabilization margin pixel number P corresponding to the level "medium" is read from the above table. is Then, an image stabilization margin with a width of 100 pixels is established around the target field of view.

[0332] Figure 12E It is shown in Figure 12A The following description assumes that the image stabilization level is "medium" (ie, the number of pixels of image stabilization margin P is 0). is is "100 pixels").

[0333] like Figure 12E As shown by the dotted line, the width of the image stabilization margin pixel number P is established around the target field of view 125 degrees. is The image stabilization margin of "100 pixels" is 126o.

[0334] Figure 12A and Figure 12EThe case where the viewing direction vi coincides with the center O of the effective projection area 122 (the optical axis center of the camera lens 16) is shown to simplify the description. On the other hand, when the viewing direction vi points to the periphery of the effective projection area 122, conversion is required to reduce the influence of optical distortion.

[0335] In step S303, the shape of the target field of view 125 established in step S301 is corrected in consideration of the viewing direction vi and the optical properties of the camera lens 16 to convert the distortion and generate the target field of view 125i. Similarly, the image stabilization margin pixel number P set in step S302 is also corrected in consideration of the viewing direction vi and the optical properties of the camera lens 16. is .

[0336] For example, the user's viewing direction should be 45° to the right of the center o, and the field angle setting value V ang In this case, the viewing direction vr is determined in step S212 (vector information (45°, 0°)), and the target field of view 125 is established as a range of 45° in each of the left, right, and top and bottom directions centered on the viewing direction vr. In addition, considering the optical properties of the camera lens 16, the target field of view 125 is corrected to Figure 12B Target field of view 125r is shown.

[0337] like Figure 12B As shown, the target field of view 125r becomes wider toward the periphery of the effective projection area 122. Furthermore, the position of the viewing direction vr is slightly inward relative to the center of the target field of view 125r. This is because the optical design of the camera lens 16 in this embodiment is similar to that of a stereoscopic fisheye lens. It should be noted that the content of the correction depends on the optical design of the camera lens 16. If the camera lens 16 is designed as an equidistant fisheye lens, an equi-stereoscopic fisheye lens, or an orthogonal fisheye lens, the target field of view 125 is corrected based on its optical properties.

[0338] Figure 12F It is shown in Figure 12B The target field of view 125r shown is given around Figure 12E Graph of an example of an image stabilization margin 126r corresponding to the same image stabilization level “Medium” as the image stabilization margin in FIG. is ) establishes an image stabilization margin 126° on the left, right, and upper and lower sides of the target field of view 125° ( Figure 12E ). In contrast, the image stabilization margin 126r ( Figure 12F ) is the number of pixels of image stabilization margin P is It is corrected to increase toward the periphery of the effective projection area 122 .

[0339] Thus, the shape of the image stabilization margin required for image stabilization around the target field of view 125r is also corrected so that, as shown in the image stabilization margin 126r, the correction amount will increase toward the periphery of the effective projection region 122. This is also because the optical design of the imaging lens 16 in the present embodiment is close to that of a stereographic fisheye lens. It should be noted that the content of the correction depends on the optical design of the imaging lens 16. If the imaging lens 16 is designed as an equidistance projection fisheye lens, an equi-solid angle projection fisheye lens, or an orthographic projection fisheye lens, the image stabilization margin 126r is corrected according to its optical properties. Figure 12F

[0340] The processing performed in step S303 of sequentially switching the shape of the target field of view 125i and its image stabilization margin in consideration of the optical properties of the imaging lens 16 is complicated processing. Therefore, in the present embodiment, the processing of step S303 is performed using a table stored in the internal nonvolatile memory 102 that holds the shape of the target field of view 125i and its image stabilization margin for each observation direction vi. It should be noted that the general control CPU 101 can have a calculation formula depending on the optical design of the imaging lens 16. In this case, the general control CPU 101 can calculate the optical distortion value using the calculation formula.

[0341] In step S304, the position and size of the image recording frame are calculated. As described above, the image stabilization margin 126i required for image stabilization is established around the target field of view 125i. However, in the case where the position of the observation direction vi is close to the periphery of the effective projection region 122, the shape of the image stabilization margin becomes quite special, for example, as shown in the image stabilization margin 126r.

[0342] The general control CPU 101 can extract an image in such a special shape region and apply a development process to the extracted image. However, a non-rectangular image is not generally used when recording as image data in step S600 or when transmitting the image data to the display device 800 in step S700. Therefore, in step S304, the position and size of the image recording frame 127i of a rectangular shape including the entire image stabilization margin 126i are calculated.

[0343] Figure 12F The image recording frame 127r calculated for the image stabilization margin 126r in step S304 is shown by a dotted line. In step S305, the position and size of the image recording frame 127i calculated in step S304 are recorded to the main memory 103.

[0344] ​In this embodiment, the upper left coordinates (Xi, Yi) of the image recording frame 127i in the ultra-wide-angle image are recorded as the position of the image recording frame 127i, and the horizontal width WXi and vertical width WYi starting from the coordinates (Xi, Yi) are recorded as the size of the image recording frame 127ii. For example, in step S305, Figure 12F The coordinates (Xr, Yr), lateral width WXr, and vertical width WYr of the image recording frame 127r are shown. It should be noted that the coordinates (Xi, Yi) are XY coordinates with the origin being a predetermined reference point (specifically, the optical center of the imaging lens 16).

[0345] When the image stabilization margin 126i and the image recording frame 127i are determined in this way, the process exits. Figure 7D The subroutine shown.

[0346] The descriptions thus far have used viewing directions with a horizontal angle of 0° (such as viewing direction v0 (vector information (0°, 0°)) and viewing direction vr (vector information (45°, 0°))) to simplify the complex optical distortion conversion. On the other hand, the user's actual viewing direction vi is arbitrary. Therefore, the following description will describe the recorded area development process performed when the horizontal angle is not 0°.

[0347] For example, in the field angle setting value V ang When is 90° and the viewing direction vm is (-42°, -40°), as Figure 12C As shown, the target field of view is 125m. In addition, the field angle setting value V ang When is 45° and the viewing direction vm is (-42°, -40°), if Figure 12D As shown in FIG, the target field of view 128m appears smaller than the target field of view 125m. Figure 12G As shown, an image stabilization margin 129m and an image recording frame 130m are established around the target field of view 128m.

[0348] Since the processing of step S400 is a basic imaging operation and adopts a general sequence of the imaging unit 40, its detailed description is omitted. It should be noted that the image signal processing circuit 43 in the imaging unit 40 in this embodiment also performs processing for converting the signal of the inherent output format (standard examples: MIPI, SLVS) output from the solid-state image sensor 42 into imaging data of a general sensor reading system.

[0349] In a case where the video image mode is selected by the imaging mode switch 12, the imaging unit 40 starts recording in response to the pressing of the start switch 14. Thereafter, in a case where the stop switch 15 is pressed, the recording ends. On the other hand, in a case where the still image mode is selected by the imaging mode switch 12, the imaging unit 40 captures a still image each time the start switch 14 is pressed.

[0350] Figure 7E is a flowchart showing a subroutine of the recording area development processing in step S500. Figure 7A In step S501, the raw data of the entire area of the imaging data (super-wide-angle image) generated by the imaging unit 40 in step S400 is obtained, and the raw data is input to the imaging unit of a head unit (not shown) called the overall control CPU 101.

[0351] In the next step S502, the portion within the image recording frame 127i is extracted from the super-wide-angle image obtained in step S501, based on the coordinates (Xi, Yi), the horizontal width WXi, and the vertical width WYi recorded in the main memory 103 in step S305. After this extraction, only the pixels within the image stabilization margin 126i are subjected to the crop development processing (S503 to S508) consisting of steps S503 to S508. Figure 7F Compared to the case where the development processing is performed on the entire area of the super-wide-angle image read in step S501, this can significantly reduce the amount of calculation. Therefore, the calculation time and the power consumption can be reduced.

[0352] As shown in Figure 7F In a case where the video image mode is selected by the imaging mode switch 12, the processing of steps S200 and S300 and the processing of step S400 are performed in parallel at the same frame rate or at different frame rates. Each time the raw data of the entire area of one frame generated by the imaging unit 40 is obtained, the crop development processing is performed based on the coordinates (Xi, Yi), the horizontal width WXi, and the vertical width WYi recorded in the main memory 103 at that time.

[0353] In a case where the crop development processing is started on the pixels within the image stabilization margin 126i, color interpolation for interpolating the data of the color pixels arranged in a Bayer arrangement is performed in step S503.

[0354] Thereafter, in step S504, the white balance is adjusted, and in step S505, color conversion is performed. In step S506, gamma correction to correct the gradation according to a gamma correction value set in advance is performed. In step S507, edge enhancement corresponding to the image size is performed.

[0355] In step S508, the image data is converted into a format that can be primarily stored by applying processing such as compression. The converted image data is stored in the main memory 103. After that, the process exits this subroutine. The details of the data format that can be primarily stored will be described later.

[0356] The order of the processes of steps S503 to S508 performed during the cropping development process may be changed according to the characteristics of the camera system. Some of these processes may be omitted. The order and presence of the processes of steps S503 to S508 do not limit the present invention.

[0357] Furthermore, in the case where the video image mode is selected, the processing of steps S200 to S500 is repeatedly performed until the recording ends.

[0358] According to this process, the amount of calculation is significantly reduced compared to the case where the development process is performed on the entire area read in step S501. Therefore, an inexpensive and low-power microcomputer can be used as the overall control CPU 101. In addition, the heat generated in the overall control CPU 101 is reduced and the life of the battery 94 becomes longer.

[0359] Furthermore, in order to reduce the control load of the overall control CPU 101, in this embodiment, the optical correction processing ( Figure 7A Step S800) and image stabilization processing ( Figure 7A Step S900) is not performed by the camera body 1. After the image is transmitted to the display device 800, these processes are executed by the display device controller 801. Therefore, if only the data of a portion of the image extracted from the projected ultra-wide-angle image is transmitted to the display device 800, the optical correction and image stabilization processes cannot be performed. In other words, since the extracted image data does not include the position information used in the formulas for the optical correction process and the correction table used for reference in the image stabilization process, the display device 800 cannot correctly perform these processes. Therefore, in this embodiment, the camera body 1 transmits correction data including information regarding the extraction position of the image from the ultra-wide-angle image, along with the extracted image data, to the display device 800.

[0360] In a case where the extracted image is a still image, since the still image data corresponds to the correction data one-to-one, even if these data are sent to the display device 800 separately, the display device 800 can correctly perform the optical correction processing and the image stabilization processing. On the other hand, in a case where the extracted image is a video image, if the video image data and the correction data are sent to the display device 800 separately, it is difficult to determine the correspondence between each frame of the video image data and the correction data. In particular, in a case where the clock rate of the general control CPU 101 in the camera body 101 and the clock rate of the display device controller 801 in the display device 800 are slightly different, during a video image capturing operation of several minutes, the synchronization between the general control CPU 101 and the display device controller 801 will be lost. This can cause a defect in which the display device controller 801 corrects a frame with different correction data from the corresponding correction data.

[0361] Therefore, in the present embodiment, when the data of the extracted video image is sent to the display device 800, the camera body 1 appropriately imparts its correction data to the data of the video image. The following describes this method.

[0362] Figure 14 is a flowchart showing a subroutine of the main recording processing in step S600 of Figure 7A . Hereinafter, this processing will be described by also referring to Figure 15 . Figure 14 Processing in a case where the video image mode is selected by the imaging mode switch 12 is shown. In a case where the still image mode is selected, the processing starts from step S601, and ends after the processing of step S606.

[0363] In step S601a, the general control CPU 101 reads the image of one frame of the video image after developed in the recording area development processing Figure 7E ), to which the processing of steps S601 to S606 is not applied. Further, the general control CPU (the metadata generation unit) 101 generates the correction data as the metadata of the read frame.

[0364] In step S601, the general control CPU 101 attaches information about the extraction position of the image of the frame read in step S601a to the correction data. The information attached in this step is the coordinates (Xi, Yi) of the image recording frame 127i obtained in step S305. It should be noted that the information attached in this step can be vector information showing the viewing direction vi.

[0365] In step S602, the overall control CPU (optical correction value obtaining unit) 101 obtains an optical correction value. The optical correction value is the optical distortion value set in step S303. Alternatively, the optical correction value can be an edge light amount correction value or a diffraction correction value corresponding to the optical properties of the lens.

[0366] In step S603, the overall control CPU 101 appends the optical correction value used for the distortion conversion in step S602 to the correction data.

[0367] In step S604, the overall control CPU 101 determines whether the image stabilization mode is effective. Specifically, in a case where the image stabilization mode is "medium" or "strong", it is determined that the image stabilization mode is effective, and the processing proceeds to step S605. On the other hand, in a case where the image stabilization mode set in advance is "off", it is determined that the image stabilization mode is not effective, and the processing proceeds to step S606. The reason for skipping step S605 in a case where the image stabilization mode is "off" is that, by skipping step S605, the amount of calculation data of the overall control CPU 101 and the amount of data of wireless communication are reduced, and the power consumption and heat generation of the camera body 1 can be reduced. Although the reduction of data used for the image stabilization processing is described, the data related to the edge light amount correction value or the data related to the diffraction correction value obtained as the optical correction value in step S602 can be reduced.

[0368] In the present embodiment, although the image stabilization mode is set in advance by the operation of the display device 800 by the user, the mode is set as the default setting of the camera body 1. Further, in a case where the camera system is configured to switch the effectiveness of the image stabilization processing after the image data is transferred to the display device 800, the processing can proceed from step S603 to step S605 directly by omitting step S604.

[0369] In step S605, the overall control CPU (movement amount detecting unit) 101 appends the image stabilization mode obtained in step S302 and the gyro data obtained during the photographing operation of the video image associated with the frame read from the main memory 813 in step S601a to the correction data.

[0370] In step S606, the overall control CPU 101 updates the video file 1000( Figure 15 ) with the data obtained by encoding the image data and the correction data. The image data corresponds to the frame read in step S606. The correction data includes various data appended in steps S601 to S605. It should be noted that, in a case where the first frame of the video image is read in step S601a, the video file 1000 is generated in step S606.

[0371] In step S607, the overall control CPU 101 determines whether the image data obtained by the recording area development process ( Figure 7E ) all frames of the video image displayed. In the case where all frames have not been read, the process returns to step S601a. ​​On the other hand, in the case where all frames have been read, the process exits the subroutine. The generated video file 1000 is stored in the internal non-volatile memory 102. In addition to the main memory 813 and the internal non-volatile memory 102, the video image may also be stored in the large-capacity non-volatile memory 51. In addition, a sending process ( Figure 7A The image file 1000 may be stored in the main memory 813 after being transferred to the display device 800.

[0372] In this embodiment, encoding means combining the image data and the correction data into one file. At this time, the image data can be compressed, or the data file formed by combining the image data and the correction data can be compressed.

[0373] Figure 15 1 is a diagram illustrating the data structure of a video file 1000. Video file 1000 consists of a header portion 1001 and a frame portion 1002. Frame portion 1002 is composed of a frame data set, each of which consists of an image of each frame and corresponding frame metadata. Specifically, frame portion 1002 includes frame data sets for the total number of frames in the video image.

[0374] In this embodiment, frame metadata is information obtained by encoding correction data, optionally supplemented with the extraction position (in-image position information), optical correction values, and gyroscope data. However, the frame metadata is not limited to this. The amount of information in the frame metadata can be varied. For example, additional information can be added to the frame metadata depending on the camera mode selected by the camera mode switch 12. Alternatively, some of the information in the frame metadata can be deleted.

[0375] The head address and offset values ​​up to each frame data set of a frame are recorded in the header 1001. Alternatively, metadata such as the time and size corresponding to the video file 1000 may be stored in the header 1001.

[0376] In master record processing ( Figure 14 ), the video file 1000 is transmitted to the display device 800. The video file 100 includes the following data sets, each of which is processed by recording area imaging ( Figure 7E) the frame of the video image being displayed and its metadata group. Thus, even in a case where the clock frequency of the overall control CPU 101 in the camera body 101 is slightly different from the clock frequency of the display device controller 801 in the display device 800, the display device controller 801 surely applies the correction processing to the video image being displayed in the camera body 101.

[0377] In the present embodiment, although the optical correction value is included in the frame metadata, the optical correction value can be assigned to the entire image.

[0378] Figure 16 is Figure 7A A flowchart of a subroutine of the transmission processing to the display device 800 in step S700 of the process. Figure 16 The processing in a case where the video image mode is selected by the imaging mode switch 12 is shown. It should be noted that in a case where the still image mode is selected, the processing starts from the processing of step S702.

[0379] In step S701, it is judged whether the video image capturing processing (step S400) using the imaging unit 40 is ended or in recording. In a case where the video image is being recorded (during the video image capturing operation), the recording area display processing (step S500) for each frame and the update of the image file 1000 in the main recording processing (step S600) (step S606) are sequentially executed. Since the power load of the wireless transmission is large, if the wireless transmission is performed in parallel during the video image capturing operation, the battery 94 needs to have a large battery capacity or a new measure against heat generation is needed. Further, from the viewpoint of the arithmetic capacity, if the wireless transmission is performed in parallel during the video image capturing operation, the arithmetic load will become large, which needs to prepare a high-spec CPU as the overall control CPU 101, thereby increasing the cost. In view of these points, in the present embodiment, the overall control CPU 101 makes the processing proceed to step S702 after the video image capturing operation ends (YES in step S701), and establishes the wireless connection with the display device 800. On the other hand, if the camera system of the present embodiment has a margin in the power supplied from the battery 94 and a new measure against heat generation is not needed, the overall control CPU 101 can establish the wireless connection with the display device 800 in advance at the time of the start of the camera body 1 or when the video image capturing operation has not started.

[0380] In step S702, the overall control CPU 101 establishes a connection with the display device 800 through the high-speed wireless communication unit 72 to transfer the large amount of data video file 1000 to the display device 800. It should be noted that the low-power wireless communication unit 71 is used to transmit a low-resolution image for checking the field angle to the display device 800 and to exchange various setting values with the display device 800. On the other hand, the low-power wireless communication unit 71 is not used for the transfer of the video file 1000 because the transmission time period becomes long.

[0381] In step S703, the overall control CPU 101 transfers the video file 1000 to the display device 800 through the high-speed wireless communication unit 72. At the end of the transmission, the overall control CPU 101 causes the process to proceed to step S704. In step S704, the overall control CPU 101 closes the connection with the display device 800 and exits the subroutine.

[0382] The case where one image file including all frames of one video image is transferred has been explained so far. On the other hand, if the recording time period of a video image is longer than several minutes, the video image can be divided into a plurality of image files by unit time. In the case where the video file has Figure 15 the data structure shown, even if one video image is transferred to the display device 800 as a plurality of image files, the display device 800 becomes usable for correcting the video image in the absence of a timing gap with the correction data.

[0383] Figure 17 is a flowchart showing a subroutine of the optical correction process in step S800 of Figure 7A The process will be explained below by also referring to 18A to 18E As described above, the process is executed by the display device controller 801 of the display device 800.

[0384] In step S801, the display device controller (video file receiving unit) 801 first receives the video file 1000 from the camera body 1 transferred in the transmission process (step S700) to the display device 800. Thereafter, the display device controller (first extraction unit) 801 obtains the optical correction value extracted from the received video file 1000.

[0385] In the next step S802, the display device controller (second extraction unit) 801 obtains the image (the image of one frame obtained by the video image capturing operation) from the video file 1000.

[0386] In step S803, the display device controller (frame image correction unit) 801 performs an optical correction process to correct the optical aberration of the image obtained in step S802 with the optical correction value obtained in step S801, and stores the corrected image into the main memory 813. An image region (extracted image region) narrower than the developed region (target field of view 125i) determined in step S303 is extracted from the image obtained in step S802, and the optical correction process is applied to the extracted image region.

[0387] 18A to 18F is a diagram for explaining the process of applying distortion correction in step S803 of Figure 17 Figure 18A is a diagram showing the position of the subject 1401 that the user looked at with the naked eye when taking an image. Figure 18B is a diagram showing the image of the subject 1401 formed on the solid-state image sensor 42.

[0388] Figure 18C is a diagram showing the developed region 1402 in the image of Figure 18B

[0389] Figure 18D is a diagram showing the extracted image obtained by extracting the image of the developed region 1402. Figure 18E is a diagram showing the image obtained by correcting the distortion in the extracted image of Figure 18D Since the extraction process is performed when the distortion of the extracted image is corrected, the field angle of the image shown in Figure 18E becomes smaller than the field angle of the extracted image shown in Figure 18D

[0390] Figure 19 is a flowchart showing the sub-routine of the image stabilization process in step S900 of Figure 7A In the following, this process will be explained by also referring to Figure 18F As described above, this process is executed by the display device controller 801 of the display device 800.

[0391] In step S901, the display device controller 801 obtains the gyro data of the current frame, the gyro data of the previous frame, and the blur amount V n-1 Det calculated for the previous frame in step S902 described below, from the frame meta data of the video file 1000. After that, the rough blur amount V n Pre is calculated from these information. It should be noted that the current frame in this embodiment is the frame under processing, and the previous frame is the immediately preceding frame.

[0392] ​​​In step S902, the display device controller 801 calculates a fine blur amount V n Det The blur amount is detected by calculating the movement amount of the feature points from the previous frame to the current frame.

[0393] The feature points can be extracted by a known method. For example, a method using a luminance information image generated by extracting only the luminance information of the image of the frame can be employed. This method subtracts an image that shifts the original luminance information image by one or several pixels from the original luminance information image. Pixels whose absolute value of the difference exceeds a threshold value are extracted as the feature points. Further, an edge extracted by subtracting an image generated by applying a high-pass filter to the above-mentioned luminance information image from the original luminance information image can be extracted as the feature points.

[0394] In the case where the luminance information images of the current frame and the previous frame are shifted by one or several pixels, the difference is calculated a plurality of times. The movement amount is obtained by calculating the position where the difference at the pixel of the feature point decreases.

[0395] Since a plurality of feature points are required as described later, it is preferable to divide each of the images of the current frame and the previous frame into a plurality of blocks, and extract the feature points for each block. The block division depends on the number of pixels and the aspect ratio of the image. In general, 12 blocks of 4*3 or 54 blocks of 9*6 are preferable. In the case where the number of blocks is too small, the trapezoidal distortion and the rotation blur around the optical axis due to the inclination of the imaging unit 40 of the camera body 1 cannot be corrected correctly. On the other hand, in the case where the number of blocks is too large, the size of one block becomes small, and thus the distance between the adjacent feature points is shortened, which leads to an error. Thus, the optimum number of blocks is selected in accordance with the number of pixels, the detection easiness of the feature points, the field angle of the subject, and the like.

[0396] Since the difference calculation is required a plurality of times in order to obtain the movement amount in the case where the luminance information images of the current frame and the previous frame are shifted by one or several pixels, the amount of calculation increases. Since the movement amount is actually calculated on the basis of the rough blur amount V n Pre and the deviation (the number of pixels) thereof, the difference calculation is performed only in the vicinity of the rough blur amount, which can significantly reduce the amount of calculation.

[0397] Next, in step S903, the display device controller 801 performs the image stabilization process using the fine blur amount V n Det obtained in step S902. Then, the process exits the subroutine. It should be noted that, as the method of the image stabilization process, the Euclidean transformation and the affine transformation that allow the rotation and the parallel translation, and the projection transformation that allows the trapezoidal correction are known.

[0398] Although the Euclidean transformation can correct the movement in the X-axis direction and the Y-axis direction and the rotation, the Euclidean transformation cannot correct the blur caused by the camera shake of the imaging unit 40 of the camera body 1 in the front-back direction or the pan and tilt direction. Therefore, in the present embodiment, the image stabilization processing is performed using the affine transformation capable of correcting the skew. The affine transformation from the coordinates (x, y) of the feature point serving as the standard to the coordinates (x', y') is represented by the following formula 100.

[0399]

[0400] If the deviation of at least three feature points is detected, the affine coefficients of the 3 x 3 matrix of the formula 100 can be calculated. However, in the case where the detected feature points are close to each other or aligned on a straight line, the image stabilization processing becomes inaccurate in the area far from the feature points or the straight line. Therefore, for the detected feature points, it is preferable to select the feature points far from each other and not on a straight line. Therefore, in the case where a plurality of feature points are detected, the feature points close to each other are excluded, and the remaining feature points are standardized by the least square method.

[0401] Figure 18F is a graph showing an image obtained by applying the image stabilization processing of step S903 to the distortion-corrected image shown in Figure 18E Since the extraction processing is performed when the image stabilization processing is performed, the field angle of the image shown in Figure 18F becomes smaller than the field angle of the image shown in Figure 18E .

[0402] A high-quality image in which the blur is corrected by performing such image stabilization processing can be obtained. The series of operations performed by the camera body 1 included in the camera system of the present embodiment and the display device 800 are described above.

[0403] When the user selects the video image mode by the imaging mode switch 12 after turning on the power switch 11 and observes the front surface in a state where the face is not rotated in the vertical direction and the horizontal direction, the face direction detection unit 20 detects the observation direction vo (vector information (0°, 0°)) as shown in Figure 12A . Thereafter, the recording direction / field angle determination unit 30 extracts the image ( Figure 12A ) in the target field of view 125o shown in Figure 11B from the super-wide-angle image projected onto the solid-state image sensor 42.

[0404] Thereafter, when the user starts observing the child (A subject 131) of Figure 11A , for example, in a state where the camera body 1 is not operated, the face direction detection unit 20 detects the observation direction vo (vector information (0°, 0°)) as shown in Figure 11CThe observation direction vm shown (vector information (-42°, -40°)). Then, the recording direction / field angle determination unit 30 extracts the image in the target field of view 125m from the ultra-wide-angle image captured by the camera unit 40 ( Figure 11C ).

[0405] Thus, in steps S800 and S900, the display device 800 applies optical correction processing and image stabilization processing to the image of the shape extracted according to the viewing direction. Thus, even if the overall control CPU 101 of the camera body 101 has low specifications, the target field of view 125m ( Figure 11C ) is converted into an obviously distorted image like Figure 11D The image shown is of a child (subject A 131) with blur and distortion corrected. That is, after the user turns on the power switch 11 and selects a mode using the image capture mode switch 12, the user can obtain an image in their own viewing direction even if the user does not touch the camera body 1.

[0406] The preset mode will be described below. Since the camera body 1 is a compact wearable device, an operation switch for changing advanced setting values, a setting screen, etc. are not installed on the camera body 1. Therefore, in this embodiment, the setting screen ( Figure 13 ) to change the advanced setting values ​​of the camera body 1.

[0407] For example, consider a scenario where a user wants to change the field angle from 90° to 45° while continuously capturing video images. In this scenario, the following operations are required. Since the field angle is set to 90° in normal video capture mode, the user performs a video capture operation in normal video capture mode, then, upon completion of the video capture operation, displays a settings screen on display device 800, and displays the settings screen to change the field angle to 45°. However, performing this operation on display device 800 during continuous capture is cumbersome and may result in the user missing the image they intended to capture.

[0408] On the other hand, in the case where the preset mode is preset to the video image capturing operation at a field angle of 45°, the user can immediately change to the zoomed-in video image capturing operation at a field angle of 45° by simply sliding the camera mode switch 12 to "Preset" after finishing the video image capturing operation at a field angle of 90°. That is, the user does not need to pause the current camera operation and perform the troublesome operation described above.

[0409] The content of the preset mode may include not only the field angle, but also the image stabilization level ("strong", "medium" or "off"), and the setting value of the voice recognition which is not described in this embodiment.

[0410] For example, when the user switches the camera mode switch 12 from the video image mode to the preset mode while continuously observing the child (A subject 131) in the previous situation, the field angle setting value V ang In this case, the recording direction / field angle determination unit 30 extracts the super wide-angle image captured by the imaging unit 40. Figure 11E The image in the target field of view 128m is shown by the dotted box.

[0411] Also in the preset mode, optical correction processing and image stabilization processing are performed in steps S800 and S900 in the display device 800. Thus, even if the overall control CPU 101 of the camera body 101 has low specifications, a high-resolution image quality can be obtained. Figure 11F The enlarged image centered on the child (A subject 131) is shown with blur and distortion corrected. ang The process in the still image mode is similar to the case where the field angle is changed from 90° to 45°. ang The field angle setting value V is 90° and the static image ang The situation of 45° is similar.

[0412] In this way, the user can obtain an enlarged image of the viewing direction in which he or she is photographed simply by switching the mode using the imaging mode switch 12 of the camera body 1 .

[0413] While this embodiment describes a case where the facial direction detection unit 20 and the imaging unit 40 are integrally formed within the camera body 1, this configuration is not limited thereto, as long as the facial direction detection unit 20 is worn on a portion of the user's body other than the head, and the imaging unit 40 is worn on the user's body. For example, the imaging / detection unit 10 of this embodiment can be worn on the shoulder or abdomen. However, if the imaging unit 40 is worn on the right shoulder, the subject on the left side is obscured by the head. In this case, it is preferable to wear multiple imaging units in a region including the right shoulder. Furthermore, if the imaging unit 40 is worn on the abdomen, spatial parallax occurs between the imaging unit 40 and the head. In this case, it is preferable to perform observation direction correction calculations to compensate for this parallax, as described in the third embodiment.

[0414] The second embodiment will be described below. In the second embodiment, the Figures 20A to 23E A method for calibrating individual differences among users wearing the camera body 1 and adjusting the differences will be described in detail.

[0415] This embodiment will be described essentially as a derivative of the first embodiment. Therefore, the same reference numerals are used for the same configurations in the camera system of the second embodiment as in the first embodiment, and duplicate descriptions are omitted. Different configurations will be described with added details.

[0416] Individual differences and adjustments made by users wearing the camera body 1 vary, including physique, the tilt angle around the neck where the camera body 1 is worn, the position of clothing such as a collar when worn, and the adjustment of the straps 82L and 82R. Consequently, the optical axis center of the camera lens 16 of the camera body 1 often does not coincide with the center of the field of view when the user is facing forward (hereinafter referred to as the "natural state"). For the user, it is preferable to align the center of the extracted recording area (target field of view 125) with the center of the field of view in the user's current posture or operation, rather than aligning the center of the recording area with the optical axis center of the camera lens 16 of the camera body 1.

[0417] Furthermore, there are individual differences not only in the center of the user's field of view in their natural state, but also in the center of the field of view depending on the head orientation (up, down, right, left, or tilted), as well as in the head's spatial movement. Consequently, individual differences also arise in the relationship between the facial orientation (viewing direction) detected by the facial orientation detection unit 20 and the center position of the target field of view 125 established based on the viewing direction (hereinafter referred to as the field of view center position). Therefore, a calibration operation is required to associate the facial orientation with the field of view center position.

[0418] Generally, it is preferred to perform the calibration operation as Figure 7A The calibration operation is part of the preparation process (step S100) in step 100. Although it is generally estimated that the calibration operation is performed when the camera body 1 is first activated, the calibration operation can be performed when a predetermined time has passed since the previous calibration or when the position of the camera body 1 relative to the user changes from the position at the time of the previous calibration. The calibration operation can be performed when the facial direction detection unit 20 becomes unable to detect the user's face. In addition, if it is detected that the user has taken off the camera body 1, the calibration operation can be performed when the user puts the camera body 1 on again. In this way, it is preferable to appropriately perform the calibration operation at the timing when it is determined that calibration is necessary in order to properly use the camera body 1.

[0419] Figure 20A and Figure 20B 1 is a diagram showing details of a calibrator 850 used in the calibration process according to the second embodiment. In this embodiment, the calibrator 850 is combined with the functions of the display device 800.

[0420] The Calibrator 850 includes Figure 1DThe A button 802, the display unit 803, the built-in camera 805, the face sensor 806, and the angular velocity sensor 807 of the components of the display device 800 shown are further included the positioning indicator 851 and the calibration button 854. In Figure 20A The B button 804 provided in the first embodiment is not shown in the present embodiment because the B button 804 is not used in the present embodiment and can be replaced with the calibration button 854 as described later.

[0421] Figure 20A The case where the positioning indicator 851 is a specific pattern displayed on the display unit 803 is shown. Figure 20B The case where the appearance of the calibrator 850 is used as the positioning indicator is shown. In Figure 20B The case where the positioning indicator center 852 described later is calculated from information about the outline of the calibrator 850.

[0422] It should be noted that the positioning indicator is not limited to Figure 20A and Figure 20B the examples. For example, the positioning indicator can be separate from the calibrator 850. The positioning indicator can be anything as long as its size is easy to measure and its shape is suitable for a user to watch. For example, the positioning indicator can be a lens cover of the imaging lens 16 or a charging unit for the camera body 1. In any case, since the basic way of thinking in the calibration operation is common, the following illustrates and mainly describes Figure 20A the calibrator 850 shown.

[0423] It should be noted that the calibrator 850 in the present embodiment will be combined with the function of the display device 800. Further, the calibrator 850 can be, for example, a dedicated device, a general-purpose smartphone, or a tablet terminal.

[0424] The positioning indicator 851 is an indicator displayed on the display unit 803 of the calibrator 850. The lateral width L851a and the vertical width L851b of the positioning indicator 851, and the positioning indicator center 852 can be calculated. Since the user faces the face toward the vicinity of the center portion of the positioning indicator 851 in the calibration processing described later, the positioning indicator 851 preferably has a shape captured at the center of the field of view. In Figure 20A The positioning indicator 851 is shown by a circle in which a cross mark and a small black circle at the center of the cross mark are arranged. However, the shape of the positioning indicator 851 is not limited to this shape. In addition to this, the positioning indicator can be a rectangular, triangular, star-shaped pattern, or a pictogram of a character.

[0425] The positioning index 851 is captured by the imaging unit 40 of the camera body 1. Based on the captured image, the display device controller (position calculation unit and distance calculation unit) 801 calculates the distance between the imaging / detection unit 10 and the calibrator 850, as well as the position coordinates of the positioning index 851 that appears in the image area. In this embodiment, the calibrator 850, which is equipped with the functions of the display device 800, performs these calculations. If the calibrator 850 does not incorporate the functions of the display device 800, these calculations are performed by the overall control CPU 101 of the camera body 101.

[0426] The angular velocity sensor 807 can measure the movement of the calibrator 850. Based on the measurement value of the angular velocity sensor 807, the display device controller 801 calculates movement information showing the position and orientation of the calibrator 850 described later.

[0427] When the user faces toward the center of the positioning index 851, the calibration button 854 is pressed. Figure 20A The middle calibration button 854 is a touch button displayed on the touch-sensitive display unit 803 , but the A button 802 or the B button 804 may be used as a calibration button.

[0428] Next, we will use Figure 21 Detailed description will be given of a flowchart of FIG. 4 for a calibration process performed when extracting an image from the ultra-wide-angle image captured by the imaging unit 40 according to the user's facial direction and when applying image processing to the extracted image.

[0429] Figure 21 8 is a flowchart showing a calibration process according to the second embodiment performed by the camera body (first calibration unit) 1 and the calibrator 805 .

[0430] For the purpose of explanation, the step in which the camera body 1 or the calibrator 850 receives the user's instruction is included in the frame in which the operating subject is the user. Figure 21 In , the steps performed by the display device controller 801 of the calibrator 850 in response to the user's instruction are included in the frame where the operating subject is the calibrator 850. Similarly, in Figure 21 In the flowchart of FIG. 1 , steps executed by the overall control CPU 101 of the camera body 1 in response to a user's instruction are included in a block in which the operating subject is the camera body 1 .

[0431] Specifically, Figure 21 The operating subject of steps S3104 and S3108 in FIG. 1 is the camera body 1. The operating subject of steps S3101, S3105, and S3106 is the user. In addition, the calibrator 850 is the operating subject of steps S3102, S3103, S3106a, S3107, S3107b, and S3110.

[0432] In this processing, in a case where the power of the calibrator 850 is not turned on, the user turns on the power of the calibrator 850 by operating the A button 802 in step S3101. Similarly, in a case where the power of the camera body 1 is not turned on, the user turns on the power of the camera body 1 by switching the power switch 11 to on. After that, the user establishes the connection between the calibrator 850 and the camera body 1. Upon establishment of the connection, the display device controller 801 and the overall control CPU 101 enter the calibration mode, respectively.

[0433] Further, in step S3101, the user wears the camera body 1, and adjusts the lengths of the belt portions 82L and 82R and the angle of the camera body 1 so that the camera body 1 will be disposed in an appropriate position and the imaging / detecting unit 10 can capture an image.

[0434] In step S3102, the display device controller (first display unit) 801 displays the positioning index 851 on the display unit 803.

[0435] In the next step S3103, the display device controller 801 informs the user that the user should hold the specified position of the calibrator 850 by indicating the display 855( Figure 22A ) to the user. In the present embodiment, five positions including the front, the upper right, the lower right, the upper left, and the lower left are sequentially designated as the specified positions. The specified positions can be set to other positions as long as the calibration is available.

[0436] In step S3104, the overall control CPU 101 activates the imaging unit 40 to enable the imaging operation, and activates the face direction detecting unit 20 to enable the detection of the face direction of the user. In step S3105, the user holds the calibrator 850 at the specified position informed in step S3103.

[0437] In the next step S3106, while maintaining the position of the calibrator 850 at the specified position, the user faces the direction of the positioning index 851 so that the center of the field of view of the user matches the positioning index 851, and presses the calibration button 854.

[0438] In step S3106a, the display device controller (second display unit) 801 determines whether or not the user is looking at the center 852 of the positioning marker 851 of the positioning marker 851, that is, whether or not the center of the field of view of the user coincides with the center 852 of the positioning marker. In the case where it is determined that the user is looking at the center 852 of the positioning marker (YES in S3106a), the display device controller 801 notifies the user of the start of calibration for the specified position by instructing the display 855 in step S3107, and re-displays the calibration button 854. In the case where the determination result in step S3106a is NO, the user repeats the process from step S3105.

[0439] In the case where the user presses the calibration button 854 in step S3107a, the display device controller 801 transmits a calibration instruction to the camera body 1 in step S3107b.

[0440] In step S3108, the overall control CPU (acquisition / detection unit) 101 acquires the super-wide-angle image including the positioning marker 851 captured by the imaging unit 40 in response to the calibration instruction from the calibrator 850, and detects the face direction using the face direction detection unit 20. Thereafter, the overall control CPU (generation unit) 101 calculates the position coordinate information with respect to the center 852 of the positioning marker in the acquired super-wide-angle image, and generates information showing the relationship between the calculated position coordinate information and the detected face direction.

[0441] Details of the process in steps S3103 to S3108 will be described below using Figures 22A to 22F . Figures 22A to 22F is a diagram for explaining a calibration operation for the front direction of the user. The calibration operation is performed so that the center position of the target field of view 125 in the image captured by the imaging unit 40 of the camera body 1 will coincide with the center position of the field of view of the user in a natural state.

[0442] Figure 22A is a diagram showing the screen displayed on the display unit 803 of the calibrator 850 in step S3103 of Figure 21 .

[0443] As shown in Figure 22A , the positioning marker 851 and the instruction display 855 indicating the position at which the user should position the positioning marker 851 are displayed on the display unit 803 of the calibrator 850.

[0444] The instruction display 855 is a display indicating the user to position the positioning indicator 851 at the center of the field of view of the user when facing the front. Note that the instruction displayed as the instruction display 855 is not limited to a string. For example, the instruction can be displayed by another method using a diagram, a picture, or a moving image, or the like. Furthermore, the instruction display 855, such as a general tutorial, can be displayed first, and then the positioning indicator 851 can be displayed.

[0445] Figure 22B is a diagram showing a state in which the user holds the calibrator 850 in the front according to the instruction displayed as the instruction display 855 in Figure 22A

[0446] In step S3105, the user holds the calibrator 850 in the front according to the instruction displayed as the instruction display 855 in Figure 22A . Then, in step S3106, the user holds the calibrator 850 so that the positioning indicator 851 will match the center of the field of view of the user when facing the front, and the user presses the calibration button 854 Figure 22A . In response to the pressing of the calibration button 854, the determination in step S3106a is made. The detailed procedure of this determination method will be described later. In the case where the determination result in step S3106a is "Yes", the display device controller 801 changes the instruction display 855 shown in Figure 22A to an instruction notifying "Start of calibration of the front direction", and displays the calibration button 854.

[0447] Then, the user presses the calibration button 854 after confirming that the instruction display 855 shown in Figure 22A is changed to an instruction notifying "Start of calibration of the front direction" (step S3107a). In response to the pressing of the calibration button 854, the calibration instruction is sent to the camera body 1 in step S3107b. And the imaging unit 40 obtains a captured image in step S3108.

[0448] Figure 22C is a diagram showing the entire super-wide-angle image captured by the imaging lens 16 in the state of Figure 22B Figure 22D is a diagram showing an image obtained by correcting the aberration of the super-wide-angle image shown in Figure 22C

[0449] Furthermore, in response to the pressing of the calibration button 854 by the user in the state of Figure 22B , the face direction detection unit 20 obtains a face direction in step S3108.

[0450] Figure 22E is a diagram showing a state in which the user holds the calibrator 850 in the front according to the instruction displayed as the instruction display 855 in Figure 21 ​​​a schematic view of the face direction image recorded by the face direction detection unit 20 in step S3108.

[0451] As described in the first embodiment using Figures 8G to 8K the chin positions 207, 207r, and 207u relative to the throat position 206, the face direction detection unit 20 calculates the angle in the lateral and vertical directions of the face. However, since the distance and angle of the chin positions 207, 207r, and 207u relative to the throat position 206, like the image center, also have individual differences and adjustment differences due to the user's constitution and the like, these distances and angles are not fixed. Therefore, in the present embodiment, the relationship between the chin position and the throat position 206 at the time when the calibration button 854 is pressed is defined as the value of the case where the user places the center of the field of view in front. This makes it possible to correctly calculate the face direction of the user regardless of the individual differences and adjustment differences.

[0452] Returning to Figure 21 In step S3109, the overall control CPU 101 determines whether the calibration of the front direction is ready. That is, it is determined whether the information required to calculate the chin position 207, the throat position 206, and the center of the positioning marker 852 has been obtained.

[0453] At this time, in the case where the acquisition of the required information is not completed, it is determined that the calibration is not ready (NO in step S3109), and the operation from step S3102 is repeated to obtain the insufficient information among the required information. In the case where the acquisition of the required information is not completed, not all of the operations from step S3102 are necessary. It is possible to only perform the operation for obtaining the insufficient information again.

[0454] The determination in step S3106a is performed using the face sensor 806 or the built-in camera 805 installed in the calibrator 850. Hereinafter, the detailed procedure of the determination method will be described using the case where the calibration operation of the front direction is performed using the built-in camera 805. Although the case where the face sensor 806 is used differs from the case where the camera is used in terms of the dimension of the information (two-dimensional information or three-dimensional information), the basic way of thinking is common. Therefore, the detailed description of the case where the face sensor 806 is used is omitted. In the case where the face sensor 806 is used in the determination in step S3106a, the face direction detection unit 20 of the camera body 1 does not perform the face detection of the user with infrared light 823 during the period in which the user is irradiated with infrared light 823 from the face sensor 806. This is to prevent the interference of the infrared light 23 and 823.

[0455] First, in step S3106, the user presses the Figure 22Awith the calibration button 854, the display device controller 801 obtains a built-in camera image 858 containing the user by capturing an image with the built-in camera (face detection unit) 805 Figure 22F Further, the display device controller 801 detects position information relating to the front of the neck 201, the chin 203, the face 204 including the nose, and the imaging / detection unit 10 (imaging unit 40) from the obtained built-in camera image 858.

[0456] In step S3106a, the display device controller (judgment unit) 801 judges whether the user is looking at the center 852 of the positioning indicator 851 at the center of the field of view using the position information detected from the built-in camera image 858.

[0457] As a result of this judgment, in a case where it is judged that the user is looking in a different direction, the display device controller 801 displays a message indicating that correct information cannot be obtained as an instruction display 855. This can instruct the user to perform the calibration operation again.

[0458] In a case where the imaging / detection unit 10 is tilted by more than a certain angle or in a case where the face direction detection window 13 is blocked or dirty, the display device controller 801 can judge that a correct calibration operation cannot be performed using the built-in camera image 858. In this case, the display device controller 801 can display a message indicating that correct information cannot be obtained as an instruction display 855.

[0459] Further, it is also possible to use the built-in camera image 858 obtained in step S3106a and the super-wide-angle image obtained in step S3108 to obtain information required for parallax correction described later in the fifth embodiment.

[0460] Specifically, before the imaging unit 40 captures the positioning indicator 851 in step S3108, information relating to the size (lateral width L851a and vertical width L851b) of the positioning indicator 851 is transmitted from the calibrator 850 to the camera body 1 in advance. Thereby, the overall control CPU 101 can calculate the distance between the imaging / detection unit 10 and the positioning indicator 851 by using the information relating to the size of the positioning indicator 851 and the image of the positioning indicator 851 appearing in the super-wide-angle image obtained in step S3108. Since the positioning indicator 851 is included in the calibrator 850 which is the same housing as the built-in camera 805, and the calibrator 850 is directly opposite the user in Figure 22B the distance between the built-in camera 805 and the imaging / detection unit 10 is equal to the distance between the positioning indicator 851 and the imaging / detection unit 10.

[0461] Similarly, in step S3106a, the built-in camera 805 is used to take a picture Figure 22F Prior to the built-in camera image shown in FIG. 8A, information about the size of the imaging / detecting unit 10 is transmitted from the camera body 1 to the calibrator 850. Thereby, the display device controller (vertical distance calculating unit) 801 can estimate the vertical distance 5070 between the optical axis center of the imaging lens 16 and the viewing position of the user by using the information about the size of the imaging / detecting unit 10 and the image of the imaging / detecting unit 10 appearing in the built-in camera image 858. In addition, the display device controller 801 can estimate the distance 2071 between the imaging lens 16 and the chin 203 of the user. The distance 2071 can be the distance between the face direction detecting window 13 and the chin 203. Figure 22F

[0462] In order for the face direction detecting unit 20 to calculate the throat position 206 and the chin position of the user, it is necessary to separate the face of the user from the face direction detecting window 13 by a distance greater than a certain distance according to the design of the face direction detecting unit 20. Therefore, this estimation result can be used as one of the judgment conditions when judging whether the face direction detecting unit 20 can correctly detect the face direction.

[0463] Returning to Figure 21 When it is judged that the required information is obtained and the preparation for the calibration of the front direction is completed, the overall control CPU 101 causes the process to proceed to step S3110.

[0464] In step S3110, the display device controller (first calibration unit) 801 calculates information required to shift the extraction center position to absorb individual differences and adjustment differences, and shifts the extraction center position based on the information.

[0465] Details of the calculation in step S3110 will be described below. If the user is in an ideal state according to the design value and wears the camera body 1 ideally, the center 856 of the super-wide-angle image obtained in step S3108 shown in FIG. 8A should almost coincide with the positioning indicator center 852 appearing in the super-wide-angle image. However, since there are actually individual differences and adjustment differences due to the user's constitution and the like, the center 856 of the super-wide-angle image is usually not coincident with the positioning indicator center 852. Figure 22C For the user, it is preferable to make the extraction center position coincide with the center of the field of view of the user in the current posture or operation (i.e., the positioning indicator center 852 in the super-wide-angle image), rather than the center 856 of the super-wide-angle image shown by the camera body 1.

[0466]

[0467] ​​Therefore, the amount of deviation of the positioning index center 852 with respect to the center 856 of the super-wide-angle image is measured, and the extraction center position is shifted to a position based on the positioning index center 852 different from the center 856 of the super-wide-angle image. Furthermore, the face direction detected by the face direction detection unit 20 at this time is also shifted in a similar manner.

[0468] A specific shifting method will be described by referring to Figure 22C and Figure 22D The amount of deviation of the positioning index center 852 with respect to the center 856 of the super-wide-angle image is measured. And as shown in Figure 22C , the measured amount of deviation is divided into a horizontal deviation amount 857a and a vertical deviation amount 857b. After performing appropriate conversion processing corresponding to the projection method of the entire field angle, the shift amount is determined based on the deviation amounts 857a and 857b.

[0469] Furthermore, as shown in FIG. 22D , the shift amount can be determined after applying appropriate conversion processing to the super-wide-angle image corresponding to the projection method. That is, the amount of deviation of the center 856a after conversion with respect to the positioning index center 852 is measured in the super-wide-angle image. And this amount of deviation is divided into a horizontal deviation amount 857c and a vertical deviation amount 857d. Then, the shift amount can be determined based on the deviation amounts 857c and 857d.

[0470] The shifting method can be arbitrarily selected from the methods shown in FIG. 22C and FIG. 22D , taking into account the processing load and the purpose of the camera system.

[0471] By performing the above-described calibration operation for the front direction, the face direction of the user wearing the camera body 1, the center of the field of view in the face direction within the super-wide-angle image, and the face direction detected by the face direction detection unit 20 are appropriately associated regardless of individual differences and adjustment differences.

[0472] Among the five directions (front, right upper, right lower, left upper, and left lower), the calibration operation for the front direction has been described so far. Similar calibration operations need to be performed for the remaining four directions.

[0473] Therefore, in the case where the processing of step S3110 in FIG. 21 is completed, the processing proceeds to step S3111. In step S3111, in the case where there is a direction among the five directions for which the calibration operation has not been performed, the target direction of the calibration operation is changed, and the processing returns to step S3103. Thereby, the calibration operation is similarly repeated for the remaining four directions other than the front direction for which the calibration operation has ended.

[0474] Although the calibration operation is performed for the front direction in FIG. 21The processing ends as is in a case where it is determined in step S3111 that there is no direction for which the calibration operation has not been performed.

[0475] FIG. 23A to FIG. 23E is a view for explaining the calibration operation for the user's right-up direction (the right-up direction in the super-wide-angle image). FIG. 23A to FIG. 23E correspond to FIG. 22A to FIG. 22E respectively, and the basic operation is also the same. Therefore, the common explanation is omitted.

[0476] As shown in FIG. 23A , the instruction display 855 displays a character string that indicates to the user to position the pointing marker 851 at the center of the field of view of the user when the face is oriented to the right-up.

[0477] FIG. 23B is a view showing a state in which the user holds the calibrator 850 to the right-up in accordance with the indication shown by the instruction display 855 in FIG. 23A . FIG. 23C is a schematic view showing the entire super-wide-angle image captured by the imaging lens 16 in the state of FIG. 23B .

[0478] As shown in FIG. 23C , first, the amount of deviation between the center 856 of the super-wide-angle image and the pointing marker center 852 is measured in accordance with a specific deviation method. Thereafter, the measured amount of deviation is divided into a radial deviation amount 857e and an angular deviation amount 857f. After performing appropriate conversion processing corresponding to the projection method for the entire angle of field, the amount of deviation is determined on the basis of the deviation amounts 857e and 857f.

[0479] Further, as shown in FIG. 23D , the amount of deviation can be determined after applying appropriate conversion processing to the super-wide-angle image corresponding to the projection method. That is, the amount of deviation of the center 856a in the super-wide-angle image after conversion with respect to the pointing marker center 852 is measured. And the amount of deviation is divided into a radial deviation amount 857g and an angular deviation amount 857h. Then, the amount of deviation can be determined on the basis of the deviation amounts 857g and 857h.

[0480] The determination of the amount of deviation described in FIG. 22A to FIG. 22E uses a method of dividing the amount of deviation into a horizontal deviation amount and a vertical deviation amount. In contrast, the determination of the amount of deviation described in FIG. 23A to FIG. 23E uses a method for dividing the amount of deviation into a radial deviation amount and an angular deviation amount. The difference in the methods is only for the sake of explanation, and any method can be used.

[0481] Further, as shown in FIG. 23EAs shown, the face direction detection unit 20 obtains the throat position 206 and the chin position 207ru required to calculate the face direction when the face is oriented upward to the right. Thus, the face direction of the user when looking in the direction toward the center 852 of the positioning indicator (in this case, the upward-to-the-right direction) can be correctly measured regardless of individual differences and adjustment differences of the user.

[0482] As described above, in the calibration process shown in FIG. 8, the calibration operation is performed for the upward-to-the-right direction in addition to the front direction. Thus, the face direction detection unit 20 can correctly detect the direction in which the user turns his or her head when the user turns his or her head in any of the up-down-right-left directions. Therefore, the user can appropriately use the camera body 1 regardless of individual differences and adjustment differences. FIG. 21

[0483] In the above description, the method of repeatedly performing the calibration operation for the five directions (front, upward-to-the-right, downward-to-the-right, upward-to-the-left, and downward-to-the-left) is described to simplify the description.

[0484] However, the calibration operation is not limited to this method. For example, the following method can be employed. That is, the user continuously moves the calibrator 850 according to the indication display 855. At the same time, the user continuously captures the positioning indicator 851 displayed on the calibrator 850 at the center of the field of view. The user moves the calibrator 850 along a zigzag trajectory, a spiral trajectory, or a polygonal trajectory, or the like. In this method, the display device controller 801 transmits the calibration indication to the camera body 1 multiple times during the movement of the calibrator 850.

[0485] Every time the calibration indication is received, the overall control CPU 101 obtains the face direction detected by the face direction detection unit 20 and the position coordinate information related to the center 852 of the positioning indicator in the super-wide-angle image captured by the imaging unit 40, and saves these as history information. Thereafter, the overall control CPU 101 calculates the relationship between the extraction center position of the image and the face direction of the user by combining the information extracted from the obtained history information. Further, in this method, the information extracted from the history information can be limited to the information obtained when the user is looking at the positioning indicator 851. This information is limited using the information related to the built-in camera 805 and the face sensor 806 obtained by the calibrator 850 during the movement of the calibrator 850. Thus, the information obtained when the user is looking elsewhere is no longer extracted from the history information, which improves the calculation accuracy of the relationship.

[0486] ​Further, the display device controller 801 can transmit the measurement value of the angular velocity sensor 807 along with the calibration instruction to the camera body 1. In this case, the overall control CPU 101 obtains movement information showing the movement track of the calibrator 850 with the user and the position and posture of the calibrator 850 from the transmitted measurement value of the angular velocity sensor 807. This movement information is also saved as history information. Thus, according to the movement information based on the measurement value of the angular velocity sensor 807, the face direction detected by the face direction detection unit 20, and the position coordinate information relating to the positioning indicator center 852 in the super-wide-angle image captured by the imaging unit 40, the calibration operation can be performed easily and correctly.

[0487] In this case, the movement information based on the measurement value of the angular velocity sensor 807 should coincide with the movement information based on the position coordinate information relating to the positioning indicator 851. Therefore, in the case of using the measurement value of the angular velocity sensor 807, it is necessary to synchronize the communication between the camera body 1 and the calibrator 850.

[0488] As described above, the second embodiment describes a calibration method capable of associating the face direction of the user with the center position of the target field of view 125 provided in the super-wide-angle image regardless of individual differences and adjustment differences. On the other hand, the present application is not limited to the various configurations exemplified in the second embodiment, and various modifications can be utilized within the scope of the present application.

[0489] Next, the third embodiment will be described. In the third embodiment, a method to prevent visually-induced motion sickness caused by an auxiliary recorded image will be described using FIG. 24A to FIG. 26

[0490] This embodiment is basically described as a derivation from the first embodiment. Therefore, the same configurations in the camera system of the third embodiment as those in the camera system in the first embodiment are denoted by the same reference numerals, and the repeated description is omitted. The different configurations will be described by adding details.

[0491] As a result of the progress of imaging technology, CG images and powerful 3D video images like the captured images can be easily enjoyed.

[0492] On the other hand, in the case of such a 3D video image being an image with dynamic movement like a VR video image or an image with large camera shake, visually-induced motion sickness often occurs when the image is watched. Since visually-induced motion sickness can cause symptoms like motion sickness, the attention to safety measures therefor is increasing.

[0493] ​The camera system should be designed so that the image in the direction in which the user's face is facing is extracted as is and imaged in the recording area development processing (step S500). In this case, in the case where the user's face moves rapidly during the imaging operation with the imaging unit 40 (step S400), the image scene also switches at a fast speed.

[0494] Although the user who moves the face rapidly during the imaging operation with the imaging unit 40 does not feel discomfort, when the image includes an image scene that moves rapidly, the viewer who appreciates the image that is assisted to be recorded in step S1000 can suffer from visually-induced motion sickness.

[0495] The above discloses a technique of capturing an image in the direction in which the face is facing, but these do not disclose a countermeasure against such visually-induced motion sickness. Therefore, the present embodiment provides a camera system that prevents the viewer from suffering from visually-induced motion sickness. Therefore, even if the user moves the face rapidly during the imaging operation with the imaging unit 40, the camera system controls so that the completed image does not include an image scene that switches at a fast speed.

[0496] As described using FIG. 8H to FIG. 8K or FIG. 10B to FIG. 10D , the user's face can turn in the up / down / right / left direction during the imaging operation with the imaging unit 40. Therefore, the direction and speed of the movement of the user's face are represented by an angular velocity ω, and the amount of movement thereof is represented by an angle θ. The angular velocity ω is calculated by dividing the angle θ detected by the face direction detection unit 20 by the detection time period.

[0497] The human action that moves the face rapidly includes looking back, glancing, moving object observation, and the like.

[0498] Looking back is, for example, an action in which the user looks back when a loud sound occurs. Glancing is an action in which a person looks at something that causes a worrisome change in the field of view once, and then returns the face to the previous position due to lack of interest. Moving object observation is an action in which a person continuously observes a moving object such as a bird and a kite flying freely in the air, and the like.

[0499] In the case where such an action occurs during the imaging operation with the imaging unit 40, and in the case where the image in the direction in which the user's face is facing is extracted as is and imaged in the recording area development processing, as described above, the viewer who appreciates the completed image can suffer from visually-induced motion sickness.

[0500] Therefore, the general control CPU 101 determines that the action that makes the face move quickly (one of the turning around, the glancing, and the moving object observation) has occurred, in a case where the general control CPU 101 keeps the state in which the angular velocity ω is equal to or greater than the threshold value ω0 for more than the first predetermined time. Further, in a case where the general control CPU 101 determines that the action that has occurred is neither the glancing nor the moving object observation, according to the method described later FIG. 25A and FIG. 25B , the general control CPU 101 determines that the action is the turning around. In this case, the general control CPU 101 does not immediately extract the image in the direction in which the face of the user is facing in the recording area visualization processing. Instead, the general control CPU 101 delays the extraction of the image with respect to the movement of the face of the user (delayed extraction).

[0501] In the present embodiment, the threshold value ω0 is set to π / 8 rad / s. This is the speed at which the face turns from the front (0°) to the right side (90°) in 4 seconds. On the other hand, the threshold value ω0 is not limited to π / 8 rad / s. For example, the threshold value ω0 can be set to (n·π) / x rad / s (x is any value) based on the frame frequency n fps.

[0502] The angular velocity ω can be calculated based on the angle θ n and the time t n at which the image of the current frame n is obtained n-1 and the angle θ n-1 and the time t n at which the image of the previous frame n-1 is obtained, by the following equation 200.

[0503]

[0504] On the other hand, the angular velocity ω can be an arithmetic average of the angular velocities from the angular velocity ω n-x of the frame n-x to the angular velocity ω n of the current frame n over x frames.

[0505] Further, in the present embodiment, although the predetermined time period is set to 0.2 seconds, the value is not limited to this.

[0506] Hereinafter, the delayed extraction in a case where the user is turning around will be described using FIG. 24A , FIG. 24B and FIG. 24C .

[0507] Although the use of FIG. 11A to FIG. 11F and FIG. 12A to FIG. 12GThe following description takes into account distortion, but in the present embodiment, distortion of the photographing lens 16 is not taken into account to simplify the description. Further, the following description assumes that the calibration processing of the second embodiment has been applied to the images of the frames, and that the center of the image of each frame coincides with the center of the field of view of the user at the time of photographing the image. Further, to describe the case where the face turns to the front side, the case where light rays within the maximum field angle of 192° are projected to the solid-state image sensor 42 is described as an example.

[0508] The region 4000 represents a pixel region that can be photographed by the solid-state image sensor 42. The image 4001( FIG. 24A ) is an image of the frame f FIG. 24B ) extracted as the target field of view 125 in the direction in which the face is currently facing. n

[0509] The image 4002( FIG. 24A ) is an image of the frame f FIG. 24B ) extracted as the target field of view 125 in the direction in which the face was facing at the previous time. n-1

[0510] Hereinafter, the value d represents the distance 4010( FIG. 24A ) from the center of the image 4002 of the frame f n-1 to the center of the image 4001 of the frame f n .

[0511] In a case where the angular velocity ω of the face based on the face direction detected by the face direction detection unit 20 is equal to or greater than a threshold value ω0, the image 4003( FIG. 24B ) is extracted as an image of the delayed extraction frame f' n from the image projected to the region 4000.

[0512] Hereinafter, the value d' represents the distance 4011 between the center of the image 4002 of the frame f n-1 and the center of the image 4003 of the delayed extraction frame f' n .

[0513] The value d" is the delayed distance 4012 from the center of the image 4001 of the frame f n to the center of the image 4003 of the delayed extraction frame f' n . At this time, the value d of the distance 4010 is greater than the value d' of the distance 4011 (d > d').

[0514] Next, the value d' of the distance 4011 is used FIG. 24C ​​To explain the method for determining the value d'. Hereinafter, a case where the user moves the face quickly from the front (observation direction vo (vector information (0°, 0°)) to the right to the right side (90°) is explained. In this case, first, the image 4021 of the frame f n extracted when the face is oriented toward the front (observation direction vo (vector information (0°, 0°)) is obtained. After a short time, the image 4022 of the frame f n+x extracted when the face is oriented toward the right to the right side (90°) is obtained.

[0515] To prevent visually-induced motion sickness, it takes at least t seconds (for example, 4 seconds) for the face to turn from the front to the right to the right side (90°). In a case where the frame frequency of the image is n fps (for example, 30 fps), the distance d' is obtained by the following equation.

[0516]

[0517] On the other hand, when the distance d" from the frame f n to the frame f' n becomes large, since the face direction is different from the recording direction, the subject that the user is looking at can not be photographed in the frame f n

[0518] In a case where the delay period becomes equal to or larger than the predetermined period Th delay (the second predetermined time), the delayed extraction is stopped, and the extraction of the direction in which the face is currently oriented (referred to as the current face direction) is started.

[0519] The delay period is the difference between the start time t0 FIG. 26 of the delay (step S4211) and the current time t n FIG. 26 at which the face is continuously moving (step S4213).

[0520] In the present embodiment, although the predetermined value Th delay is set to 1 second, it is not limited to 1 second. For example, the predetermined value Th delay may be set to 20 / n seconds based on the frame frequency n fps. In a case where the predetermined value Th delay is 20 / n seconds, as the frame frequency becomes higher, the predetermined value Th delay becomes shorter. Since the possibility of visually-induced motion sickness becomes lower as the frame frequency becomes higher, it is possible to return the processing to the extraction of the current face direction in a short delay period.

[0521] ​​On the other hand, in a case where the delayed extraction is stopped and the extraction of the current face direction is restarted, the image scene is suddenly switched. Since such a sudden switching of the image scene causes an unnatural feeling to the user, an image effect such as fade-out and fade-in can be employed.

[0522] Further, the trajectory of the current face direction is saved so that the extraction of the current face direction can be restarted. The face direction trajectory saved in a case where it is judged that the user is glancing is explained using FIG. 25A

[0523] In a case where the delay period becomes equal to or greater than the predetermined value Th delay , the processing for stopping the delayed extraction and restarting the extraction of the current face direction is executed. In addition, in a case of glancing (i.e., in a case where the user changes the face direction to a certain direction once and immediately returns to the previous direction), this processing is also executed.

[0524] FIG. 25A is a diagram showing an example of the trajectory of the face direction in a case where the user is glancing. The center positions 4101 to 4107 of the images of the frames f n-3 to f n-2 to f n-1 and f n are positions of the center of the field of view of the user at the time when the face movement starts. Hereinafter, such a movement of the center of the field of view of the user will be referred to as a face motion vector.

[0525] The center of the field of view of the user temporarily stops at the center position 4104 and then moves to the center positions 4105, 4106 and 4107 of the images of the frames f nx+1 to f nx+2 and f nx+3 , and stops at the center position 4107 of the image of the frame f nx+3 . That is, the direction of the face motion vector from the position 4101 to the position 4104 is opposite to the direction of the face motion vector from the position 4104 to the position 4107.

[0526] In a case where the frame groups in which the motion vectors are opposite to each other are detected as FIG. 25A illustrated above, the overall control CPU 101 judges that these frame groups correspond to glancing.

[0527] In this case, the overall control CPU 101 performs the delayed extraction from the position 4101 at which the face starts to move until the position 4104 at which the motion vector starts to move in the opposite direction. This is because the position 4101 is considered to be a position of the subject which the user wants to glance at.

[0528] ​On the other hand, after the delay extraction up to the position 4101 is performed, the overall control CPU 101 stops the delay extraction, and restarts the extraction of the current face direction up to the position 4107 at which the face movement stops.

[0529] Further, in a case where the body is detected near the center of the field of view in the face direction during the movement of the user's face, and in a case where the body keeps its position near the center of the field of view in the face direction, the overall control CPU 101 judges that the user is observing the moving object. In this case, the delay extraction is not performed in the present embodiment.

[0530] FIG. 25B is a graph showing an example of the image of the frame in which the user is observing the moving object. The frame f n-3 The center position of the image 4121 coincides with the center of the user's field of view at the start of the face movement. Thereafter, the center of the user's field of view moves to the center positions of the images 4122, 4123, 4124, 4125, and 4126 of the frames f n , f n+1 , f n+2 , f n+3 , and f n+4 .

[0531] In a case where the same subject keeps its position near the center of the images of the consecutive frames as exemplified in FIG. 25B , the overall control CPU 101 judges that these frames belong to the frame group of the moving object observation.

[0532] In this case, the overall control CPU 101 does not perform the delay extraction. This is because the delay extraction during the moving object observation increases the possibility that the subject is not captured in the image.

[0533] Further, when the appreciator appreciates the video of the images 4121 to 4126 extracted in response to the rapid movement of the user's face during the moving object observation, the appreciator can suffer from the visually-induced motion sickness. Therefore, the overall control CPU 101 does not perform the image extraction related to the frame group of the moving object observation, and records the image of the entire pixel region 4000 that can be captured by the solid-state image sensor 42.

[0534] It should be noted that a margin called a blind zone can be given to the threshold value ω0, the predetermined time period, and the predetermined value Th delay .

[0535] Next, the visually-induced motion sickness prevention process according to the present embodiment will be explained using the flowchart of FIG. 26 . It should be noted that this process is executed every time the imaging unit 40 captures the frame image in step S400 during the video image capturing operation.

[0536] In step S4201, the overall control CPU 101 obtains the face direction (viewing direction) recorded on the main memory 103 in the face direction detection processing performed for the current frame imaging operation.

[0537] In step S4202, the overall control CPU 101 obtains the position and size of the image recording frame (extraction region) recorded on the main memory 103 in the recording direction / region determination processing performed for the current frame imaging operation.

[0538] In step S4203, the overall control CPU (computing unit) calculates the angular velocity ω of the face on the basis of the face direction of the current frame imaging operation obtained in step S4201, the face direction of the previous frame imaging operation stored in the main memory 103, and the frame frequency. Thereafter, the overall control CPU 101 determines whether the face has started moving at an angular velocity ω exceeding the threshold value ω0.

[0539] Specifically, when the face of the user has started moving at an angular velocity ω exceeding the threshold value ω0 for a predetermined period of time (0.2 seconds), the overall control CPU 101 determines that the face has started moving at an angular velocity ω exceeding the threshold value ω0. In the case where it is determined that the face has started moving (YES in step S4203), the processing proceeds to step S4204. Otherwise (NO in step S4203), the processing returns to step S4201. That is, even if the face of the user moves at an angular velocity ω exceeding the threshold value ω0, in the case where the period of time is less than the predetermined period of time (less than the first predetermined period of time), the processing returns to step S4201. Further, in the case where the angular velocity of the face cannot be calculated in step S4203 because the face direction at the previous frame imaging operation is not saved in the main memory 103, the processing returns to step S4201.

[0540] In step S4204, the overall control CPU 101 determines whether the face has moved by more than a predetermined angle on the basis of the angular velocity ω of the face calculated in step S4203. In the case where it is determined that the face has moved (YES in step S4204), the processing proceeds to step S4206. Otherwise (NO in step S4204), the processing proceeds to step S4205. It should be noted that in step S4204, the overall control CPU 101 can determine whether the face has moved at an angular velocity exceeding a predetermined angular velocity for a predetermined period of time (0.2 seconds).

[0541] In step S4205, the overall control CPU 101 judges whether or not the movement of the face is stopped based on the angular velocity ω of the face calculated in step S4203. In the case where it is judged that the movement is stopped (YES in step S4205), the processing returns to step S4201. Otherwise (NO in step S4205), the processing returns to step S4204.

[0542] In step S4206, the overall control CPU 101 judges whether or not the photographed subject is moving, that is, judges whether or not the user is observing a moving object. In the case where it is judged that the subject is moving (YES in step S4206), the processing proceeds to step S4207. Otherwise (NO in step S4206), the processing proceeds to step S4208.

[0543] In step S4207, the overall control CPU 101 determines that the crop development processing is not performed in the recording area development processing of the current frame, but the development processing of the entire area RAW data obtained from the entire area of the solid-state image sensor 42 is performed. Then, the processing proceeds to step S4205.

[0544] In step S4208, the overall control CPU 101 stores the face direction at the time of the current frame imaging operation obtained in step S4201 to the main memory 103. Then, the processing proceeds to step S4209.

[0545] In step S4209, the overall control CPU (delay unit) 101 determines that, in the recording area development processing of the current frame, the crop development processing (delay extraction) is performed with respect to the extraction area centered on the position shifted by the distance d with respect to the face direction of the previous frame. After that, the processing proceeds to step S4210.

[0546] In step S4210, the overall control CPU 101 judges whether or not the start time t0 of the time period stored in the main memory 103 is cleared. In the case where it is judged that the start time is cleared (YES in step S4210), the processing proceeds to step S4211. Otherwise (NO in step S4210), the processing proceeds to step S4212.

[0547] In step S4211, the overall control CPU 101 stores the current time as the start time t0 to the main memory 103. Then, the processing proceeds to step S4212.

[0548] In step S4212, the overall control CPU 101 judges whether or not the movement of the face reaches a predetermined value Th delayThe processing proceeds to step S4215. Otherwise (NO in step S4212), the processing proceeds to step S4213.

[0549] In step S4213, the overall control CPU 101 calculates a delay time period by subtracting the start time to from the current time t n stored in the main memory 103. Then, the processing proceeds to step S4214.

[0550] In step S4214, the overall control CPU 101 calculates a delay time period by subtracting the start time to from the current time t n stored in the main memory 103, and judges whether the delay time period is equal to or greater than a predetermined time period Th delay . In the case where the delay time period is equal to or greater than the predetermined time period Th delay (step S4214: YES), the processing proceeds to step S4215. Otherwise (step S4214: NO), the processing returns to step S4206.

[0551] In step S4215, the overall control CPU 101 clears the start time to stored in the main memory 103. Then, the processing proceeds to step S4216. In step S4216, the overall control CPU 101 determines the recording direction and the field angle by the recording direction / field angle determination unit 101 based on the face direction detected by the face direction detection unit 20. Then, the processing proceeds to step S4217.

[0552] In step S4217, the overall control CPU 101 sets a flag to the metadata of the current frame. Then, the processing returns to step S4201. The flag set to the metadata is used to determine the timing of adding an image effect (such as the fade-in and fade-out described above, etc.) at the time of the auxiliary recording processing described in step S1000 of the first embodiment.

[0553] As described above, in the present embodiment, in the case where the angular velocity ω of the face becomes to exceed the threshold value ω0, the frame in the face direction is not extracted as it is, but the frame is extracted in accordance with the movement of the face. This has the effect of mitigating visually-induced motion sickness.

[0554] Next, the fourth embodiment will be described. The fourth embodiment describes how to correct the extraction region of an image in accordance with the moving speed of the orientation of the face of the user using FIG. 27A to FIG. 27F , FIG. 28A and FIG. 28B .

[0555] This embodiment is basically explained as a derivative from the first embodiment. Therefore, the same configurations in the camera system of the fourth embodiment as those of the camera system in the first embodiment are denoted by the same reference numerals, and the repeated explanation is omitted. The different configurations will be explained by adding details.

[0556] First, the human action of changing the observation direction will be explained. Generally, when a person finds an item of interest in a peripheral region of the field of view that deviates from the center of the field of view, and turns the observation direction to the item, the face first moves, and the body follows after the movement of the face exceeds a certain amount.

[0557] That is, in this case, the direction of the imaging / detecting unit 10 (16) of the clavicle does not change only when the face is changing the orientation in the initial movement. After that, when the user starts to change the orientation of the entire body, the direction of the imaging lens 16 of the camera body 1 also moves. The following explanation presupposes this characteristic feature of the human action. FIG. 10A

[0558] Further, in a case where the face direction detecting unit 20 detects the face direction, a change due to a detection error occurs. In a case where the extraction position of the image is calculated based on the detection result of the face direction including the change, blurring like a result of camera shake occurs in the video image recorded in step S1000, which deteriorates the appearance. Therefore, the slight change is eliminated by applying a low-pass filter to the detection result of the face direction to correct the slight detection shake.

[0559] Further, if the face direction is detected after the instantaneous movement (for example, left and right checks while walking along a public road), the video image recorded in step S1000 tends to cause visually-induced motion sickness. Therefore, in this embodiment, a process for removing (smoothing) the slight movement component of the face direction detected by following the instantaneous movement for about 1 to 2 seconds is employed. Thereby, the appearance of the video image recorded in step S1000 is improved.

[0560] Next, the outline of the extraction region correction process in this embodiment will be explained using FIG. 27A to FIG. 27F

[0561] FIG. 27A to FIG. 27F The horizontal axis of each graph shown in FIG. 10 indicates the elapsed time. FIG. 27A The vertical axis in FIG. 10 indicates the angle movement of the actual observation center. FIG. 27B The vertical axis in FIG. 11 indicates the angle of the face direction. FIG. 27C The vertical axis in FIG. 12 indicates the angle of the direction of the imaging lens 16. Also FIG. 27D The vertical axis in FIG. 12 indicates the angle of the direction of the imaging lens 16. Also FIG. 27E The vertical axis in FIG. 12 indicates the angle of the direction of the imaging lens 16. Also FIG. 27F ​​The vertical axis in represents the angle of the extraction position. It should be noted that the upward direction on the vertical axis shows the right direction.

[0562] FIG. 27A It is a graph showing the movement of the actual observation center (face direction). FIG. 27A The angle of the vertical axis of represents the user's facial direction relative to a fixed place such as the ground (ground reference), and does not represent the angle showing the facial direction detected by the facial direction detection unit 20. That is, FIG. 27A The graph in shows that the user initially faces forward and begins to turn his face to the right at approximately 1 second.

[0563] FIG. 27B is a graph showing the detection result (viewing direction vi) of the face direction detection unit 20. FIG. 27B The reason why the line of the detection result of is not smooth is that, as described above, the detection result contains variations due to detection errors. Therefore, in this embodiment, a low-pass filter is applied to the detection result of the face direction detection unit 20.

[0564] Furthermore, processing is performed for removing (smoothing) rapid changes in the face direction detected by following the instantaneous movement of the face. FIG. 27B No such rapid changes are shown.

[0565] FIG. 27C is shown by applying a low-pass filter to FIG. 27B The smoothed result obtained by the detection result of the face direction detection unit 20 is shown in FIG. FIG. 27C As shown, by applying a low-pass filter, FIG. 27B On the other hand, as a result of applying this filter, the line of the detection result becomes a smooth line. FIG. 27C The face is detected turning from the front to the right at about 2 seconds. FIG. 27C In the curve diagram relative to FIG. 27A The movement directly corresponds to FIG. 27B It should be noted that the curve of FIG. 27B and FIG. 27C The angle of the vertical axis shows the angle relative to the direction of the imaging lens 16 (taking the camera body 1 as a reference), rather than FIG. 27A The angle of the ground reference in .

[0566] In addition, FIG. 27B In, with FIG. 27A In contrast, the inclination becomes gentle from about 4 seconds. This means that due to FIG. 27DThe camera body 1 (direction of the imaging lens 16) starts to move along with the user's body from about 4 seconds onward, so the moving speed of the face direction detected by the face direction detection unit 20 is relatively decelerated.

[0567] FIG. 27E The result of a simple addition method, which calculates the extraction position (i.e., the observation direction that is the center of the target field of view 125) by adding the amount of movement of the camera body ( FIG. 27D ) to the face direction detection result ( FIG. 27C ) smoothed by applying a low-pass filter, is shown. However, in the case where the extraction position is calculated by this simple addition method, the cropping position does not follow the movement of the actual observation center. Therefore, the video image completed in the assist recording processing includes a scene in which the pan suddenly accelerates from about 4.5 seconds onward from the start of the movement of the body.

[0568] That is, in order to eliminate the sense of discomfort with the movement of the actual observation center, it is preferable to calculate the extraction position (desired value) so as to keep the pan substantially constant as FIG. 27F indicated.

[0569] Therefore, in the present embodiment, the extraction position is calculated to avoid a scene in which the pan suddenly accelerates as FIG. 27E indicated. In the case where there are two moving speeds (0° / s and 10° / s) of the extraction position as FIG. 27F indicated, the desired value as FIG. 27D indicated is calculated by adding the amount of movement of the camera body 1 in FIG. 27C to the face direction detection result in FIG. 27F with a time lag (1 second in the present embodiment) in advance. In fact, the moving speed of the extraction position is not limited to the above two, but gradually changes. That is, the observation direction does not suddenly accelerate and does not suddenly stop. The deceleration is gradual. However, the desired value cannot draw a gradually decreasing curve according to the above calculation method. Therefore, in the present embodiment, in the case where the movement of the camera body 1 stops, the moving speed of the extraction position in the period from the start of the movement of the observation direction to the stop or in the past certain period is distributed between several frames so that the desired value will draw a gradually decreasing curve.

[0570] After that, the flowcharts of FIG. 28A and FIG. 28B will be used to sequentially explain the extraction area correction processing in the present embodiment. Hereinafter, the explanation about the same parts as those of the first to third embodiments will be simplified or omitted.

[0571] FIG. 28A is a flowchart showing a subroutine of the recording direction / area determination processing in step S300 of FIG. 7A according to the present embodiment.

[0572] In step S4000a, the observation direction vi obtained through the face direction detection processing of step S200 is smoothed using a low-pass filter (smoothing unit). As described above using FIG. 27B the observation direction vi has a change due to some detection error. The low-pass filter takes a simple moving average of the past several times (for example, 5 to 10 times). At this time, as the number of times of averaging increases, the tracking delay at the time of face direction movement becomes greater. Further, in a case where the user turns the face to the right and immediately to the left, it can not be possible to detect the observation direction vi at the time of turning to the far right.

[0573] Further, since the mixed state of the detection error depends on the detection method, the degree of smoothing can be changed according to the detection method. The application method of the low-pass filter in the vertical direction can be changed from the application method in the lateral direction.

[0574] Further, in many cases, from the viewpoint of storing the user experience as an image, it is not necessary to record the instantaneous movement of the face. For example, the user has no choice but to check the safety of the left and right while walking as described above. It is not necessary to record the image photographed at such a moment. Therefore, in the present embodiment, in step S4000a, the observation direction vi obtained when the moved observation direction returns to the previous direction within about 2 seconds is also smoothed.

[0575] Although safety checks in the left and right directions and the lower direction are necessary in many cases, safety checks in the upward direction are rarely necessary. Therefore, the low-pass filter can not be applied to the upward movement.

[0576] In a case where the extraction region is determined through the processing of steps S301 to S304, FIG. 7D the overall control CPU (second calibration unit) 101 causes the processing to proceed to step S4000 and performs the extraction region correction processing.

[0577] After that, in step S305, the extraction region after the correction is recorded, and the processing exits the subroutine. The extraction region correction processing is explained using the flowchart of FIG. 28B .

[0578] FIG. 28B is a flowchart showing the extraction region correction processing in step S4000. In FIG. 28B step S4001, the overall control CPU (movement speed calculation unit) 101 obtains the gyro information (i.e., the movement of the camera body 1 in the current frame (gyro movement amount)) from the angular velocity sensor 107.

[0579] Although the angular velocity sensor 107 is used in the present embodiment, another sensor can be used as long as the movement of the camera body 1 can be detected. For example, a magnetometric sensor (not shown) that measures the magnitude and direction of a magnetic field can be used, and an acceleration sensor 108 that detects acceleration can be used. Furthermore, a method that extracts feature points, detects a motion vector by calculating the movement amount of the feature points, and calculates the movement amount of the camera body 1 can be used. The feature points can be extracted by a known method. For example, the movement amount can be calculated by subtracting a plurality of edge images in a bias state that are extracted by applying a band-pass filter to an image obtained by extracting only luminance information from two images to calculate a position where the difference becomes small. Although this method increases the amount of calculation, since hardware such as the angular velocity sensor 107 becomes unnecessary and the weight of the camera body 1 can be reduced, this is one of the preferable aspects.

[0580] The following describes a case where the gyro information is obtained from the angular velocity sensor 107. In step S4002, the movement speed of the camera body 1 (gyro movement speed) is calculated from the gyro information obtained in step S4001 and the past obtained gyro information.

[0581] In step S4003, it is determined whether the gyro movement speed calculated in step S4002 is decelerating. In the case where the movement speed is not decelerating (NO in step S4003), the process proceeds to step S4004. Otherwise (YES in step S4003), the process proceeds to step S4006.

[0582] In step S4004, the general control CPU (second calibration unit and observation direction correction unit) 101 calculates the movement speed of the extraction position (extraction position movement speed) from the extraction position determined in step S304 and the past obtained extraction position. Next, the general control CPU 101 obtains a subtraction amount by subtracting the gyro movement speed obtained at a timing advanced by a time lag caused by applying a low-pass filter from the calculated extraction position movement speed.

[0583] In step S4005, the general control CPU 101 stores the extraction position movement speed and the subtraction amount obtained in step S4004 to the main memory 103. Then, the process exits the subroutine.

[0584] In step S4006, the total control CPU 101 calculates the expected value by distributing the total of the subtraction amount stored in the main memory 103 among the extraction position moving speeds stored in the main memory 103 so that the change in the extraction position moving speed over a certain period of time in the past becomes constant. Then, the process exits this subroutine. The certain period of time in the past can be a period of time from the start of the actual movement of the extraction position until now, or can be a period of time from when the angular velocity sensor 107 detects the movement of the camera body 1 until now. Further, in order to simplify the process, the certain period of time in the past can be a fixed period of time of 0.5 to 3 seconds. Note that the expected value before the certain period of time in the past is set to the extraction position moving speed obtained in step S4004.

[0585] Table 1 below shows FIG. 27A to FIG. 27F the changes in the data (speed) of the graphs shown. That is, the extraction position moving speed determined in step S304 is shown in row C of Table 1. The gyro moving speed calculated in step S4002 is shown in row D of Table 1. Further, the expected value calculated in step S4006 is shown in row E of Table 1.

[0586] Table 1

[0587]

[0588] In the following, the subroutine of the face region correction process will be explained as shown in Table 1 with respect to a case where the user first faces the front and gradually turns the face to the right. FIG. 28B

[0589] Since the user looks at the front at the start, the gyro moving speed calculated in step S4002 becomes about 0° / s. That is, it is determined in step S4003 that the gyro moving speed is not decelerating, and the process proceeds to step S4004. In this case, since the position of the face does not change, the extraction position moving speed also becomes 0° / s. Further, the subtraction amount calculated in step S4004 also becomes 0° / s.

[0590] Although the user starts to turn the face to the right at about 1 second, the extraction position moving speed still remains 0° / s as shown in FIG. 27C due to the time lag caused by the low-pass filter. On the other hand, since the camera body 1 does not move, the gyro moving speed is about 0° / s as shown in FIG. 27D . Therefore, as when the user still faces the front, the subtraction amount calculated in step S4004 also becomes 0° / s.

[0591] When the user further turns the face to the right at about 2 seconds, the extraction position moving speed becomes about 0.5° / s as shown in FIG. 27C ​As shown, the extracted position moving speed becomes 10° / s. On the other hand, since the camera body 1 does not move, the actual angular velocity of the face becomes 0° / s as shown. FIG. 27D As shown, the gyro moving speed is about 0° / s. Therefore, the subtraction amount calculated in step S4004 becomes 10° / s.

[0592] When the user further turns the face to the right at about 4 seconds, the user's body starts to turn to the right. That is, since the direction of the camera body 1 changes as shown, FIG. 27D the gyro moving speed becomes 10° / s. Since the user's body starts to turn, the actual angular velocity of the face decelerates the relative speed between the camera body 1 and the face direction as shown. FIG. 27B On the other hand, since the time lag due to the low-pass filter, FIG. 27C the extracted position moving speed shown in FIG. 8 still remains 10° / s. Therefore, the subtraction amount calculated in step S4004 becomes 10° / s by taking the time lag into account.

[0593] When the user further turns the face to the right at about 5 seconds, the gyro moving speed still remains 10° / s FIG. 27D ). On the other hand, FIG. 27C the extracted position moving speed shown in FIG. 8 decelerates and becomes 0° / s. Therefore, the subtraction amount calculated in step S4004 becomes -10° / s.

[0594] When the user ends the right turn action after 6 seconds FIG. 27A to FIG. 27F , the gyro moving speed becomes 0° / s, and in this case, the process is allowed to proceed to step S4006. In this case, the total of the subtraction amounts calculated so far and stored in the main memory 103 becomes +10° / s. The desired value is calculated by allocating the total of the subtraction amounts so that the change in the extracted position moving speed stored in the main memory 103 over a certain period of time in the past will become constant. In this case, as shown in Table 1, the extracted position moving speed shown in FIG. 8 is 10° / s, 10° / s, 10° / s, and 0° / s. Therefore, all the desired values in the period of time from 2 seconds to 6 seconds are set to 10° / s to keep the change in the extracted position moving speed constant (no change in this embodiment) in this period of time. FIG. 27C

[0595] ​Although the data is described in units of seconds in this embodiment to simplify the explanation, the frame rate of the video image capturing operation is usually 24 to 60 fps. On the other hand, since the face direction detection processing and the extraction region correction processing are not required to be performed at 60 times per second in many cases, the timing at which the face direction detection processing and the extraction region correction processing are performed is preferably changed from the imaging timing. For example, even when the frame rate of the video image capturing operation is 60 fps, the timing at which the face direction detection processing and the extraction region correction processing are performed can be set to 10 fps. The timing can be appropriately changed in consideration of the use, power consumption, and the like.

[0596] As described above, this embodiment shows an example that keeps the moving speed of the viewing direction constant to avoid an undesirable appearance of the video image due to a change in the moving speed of the field of view caused by combining the movement of the face and the movement of the user's body (camera body) during a large movement of the viewing direction.

[0597] Although this embodiment shows an example of cropping the ultra-wide-angle image according to the viewing direction, the present application is not limited to this. For example, the overall control CPU (imaging direction changing unit) 101 can change the imaging direction of the imaging unit 40 according to the viewing direction. In this case, the camera body 1 needs to provide a mechanism (driving mechanism) that mechanically changes the imaging direction of the imaging unit 40, specifically the directions of the imaging lens 16 and the solid-state image sensor 42, in the yaw direction and the pitch direction.

[0598] In addition, in the case where the overall control CPU (image stabilizing unit) 101 performs the image stabilizing processing described in the first embodiment, since the tracking of the face direction is delayed due to the image stabilizing processing, it is preferable to perform the processing of smoothing the face direction detection result shown in this embodiment.

[0599] Next, the fifth embodiment will be described. The fifth embodiment uses FIG. 29A to FIG. 34C to describe a method for reducing the difference between the user's field of view and an auxiliary recording image (hereinafter referred to as "recording image") caused by the parallax between the user's eye position and the wearing position of the imaging / detection unit 10.

[0600] This embodiment is basically described as a derivative from the first embodiment. Therefore, the same configurations in the camera system of the fifth embodiment as those in the camera system in the first embodiment are denoted by the same reference numerals, and the repeated explanation is omitted. The different configurations will be described by adding details. For the support of understanding, first, the difference between the user's field of view and the recording image will be described.

[0601] FIG. 29A and FIG. 29Bis a schematic view for explaining a relationship between a field of view of the user 5010 and a target field of view in a case where the close-range subject is the observation object 5020 in the first embodiment.

[0602] FIG. 29A is a schematic view showing an image 5900 including the observation object 5020 captured by the solid-state image sensor 42. FIG. 29B is a schematic view showing a positional relationship between the user 5010 and the observation object 5020.

[0603] As shown in FIG. 29B , in a case where the observation object 5020 is lower than the height of the eyes 5011 of the user, the face direction 5015 of the user 5010 is turned downward. At this time, the observation object 5020 having a background like a floor (not shown) is captured in the field of view of the user 5010.

[0604] In the first embodiment, an observation direction 5040( FIG. 29B ) parallel to the face direction 5015 of the user detected by the face direction detection unit 20 is set as the recording direction. Therefore, in a case where the short-range subject is the observation object 5020 as shown in FIG. 29B , a region 5045 not including the observation object 5020 is set as the target field of view.

[0605] In this case, even if a background (for example, a ceiling (not shown)) captured by the imaging / detection unit 10 will be different from a background (for example, a floor (not shown)) of the field of view of the user 5010, the observation direction should be set to the direction 5030 so that a region 5035 including the observation object 5020 will be the target field of view.

[0606] The above problem is caused by a parallax due to a difference between the position of the eyes 5011 of the user 5010 and the wearing position of the imaging / detection unit 10. Therefore, in the present embodiment, a parallax correction mode process for appropriately adjusting the recording direction set based on the face direction of the user 5010 corresponding to the parallax is executed.

[0607] FIG. 31 is a block diagram showing a hardware structure of the camera body 1 according to the present embodiment. The hardware structure of the camera body 1 in the present embodiment is different from the hardware structure of the camera body 1 in the first embodiment shown in FIG. 5 in that a distance sensor 5100 is added. In the present embodiment, as shown in FIG. 30 , the distance sensor 5100 is provided at the outer edge of the stop switch 15. However, the mounting position of the distance sensor 5100 is not limited to a particular position.

[0608] The distance sensor 5100 measures the distance to the subject. It should be noted that the structure of the distance sensor 5100 is not particularly limited. In this example, the distance sensor 5100 is an active type sensor that projects infrared light, laser light, millimeter waves, or the like to the subject and measures the distance to the subject by receiving the reflection thereof. Further, the distance sensor 5100 can be a passive type sensor that measures the distance to the subject based on the phase difference of incident light that passes through the photographing lens 16. The distance sensor 5100 is connected to the overall control CPU 101 and is controlled by the overall control CPU 101.

[0609] FIG. 32A and FIG. 32B is a schematic diagram for explaining the relationship between the user, the calibrator 850, and the target field of view 5080 during the calibration processing including the parallax correction mode processing in the present embodiment. FIG. 32A is a schematic diagram showing an image 5900 including the calibrator 850 that is captured by the solid-state image sensor 42. FIG. 32B is a schematic diagram showing the positional relationship between the user 5010 and the calibrator 850.

[0610] FIG. 32A The target field of view 5080 in

[0611] On the other hand, FIG. 32A The target field of view 5090 in

[0612] FIG. 33A is a flowchart showing the parallax correction mode processing as a part of the preparation processing in step S100 of FIG. 7A In the following, the details of this processing will also be explained by using FIG. 32A and FIG. 32B

[0613] In the preparation processing of step S100 of FIG. 7A When the parallax correction mode is started by the operation of the calibrator 850 by the user 5010 (step S5101), the display device controller 801 displays the positioning indicator 851 (step S5102).

[0614] Subsequently, the display device controller 801 specifies the position at which the user should hold the calibrator 850 (the specified position). Specifically, the display device controller 801 gives an instruction to the user to hold the calibrator 850 at a position at which the user can see the positioning indicator 851 in the display 800 (step S5103). FIG. 22A ​The illustrated instruction display 855 displays a similar instruction display to instruct the user 5010 to position the positioning indicator 851 at eye level in the front (step S5103).

[0615] After checking the instruction display, the user 5010 holds the calibrator 850 at the specified position specified in step S5103, and faces the face direction 5015 toward the positioning indicator 851 (front). At this time, the user 5010, the positioning indicator 851, and the imaging / detecting unit 10 constitute FIG. 32B the illustrated positional relationship.

[0616] Thereafter, in a case where it is determined that the user is looking at the positioning indicator center 852 at the center of the field of view, the display device controller 801 measures the distance 5050 (step S5104) between the imaging / detecting unit 10 and the positioning indicator 851 with the distance sensor 5100. FIG. 32B

[0617] Subsequently, the general control CPU 101 detects the horizontal axis 5060 of the imaging / detecting unit 10 with the angular velocity sensor (attitude detecting unit) 107 (step S5105). By this, the horizontal position 5065 of the image 5900 (step S5106) taken by the solid-state image sensor 42 is specified. FIG. 32A

[0618] Further, in step S5105, the general control CPU 101 obtains the distance 5855 (step S5106) between the center of the positioning indicator 851 on the image 5900 and the horizontal position 5065. Thereafter, the general control CPU (angle calculating unit) 101 calculates the angle 5055 (step S5106) between the horizontal axis 5060 and the direction of the positioning indicator 851 as seen from the imaging / detecting unit 10. This calculation is performed using the distance 5855 and information about the relationship between a point on the image 5900 and the angle of incidence of a light ray imaged at the point. This information is stored in a memory (for example, the internal nonvolatile memory 102). FIG. 32A FIG. 32B Thereafter, in step S5106, the general control CPU (vertical distance calculating unit) 101 calculates the vertical distance 5070 between the imaging / detecting unit 10 and the eyes 5011 of the user 5010 using the distance 5050 and the angle 5055 calculated in step S5105. Then, the process exits this subroutine.

[0619]

[0620] ​​​​In the present embodiment, the vertical distance 5070 between the imaging / detecting unit 10 and the eye 5011 of the user 5010 is measured by a method different from that of the second embodiment. However, the measurement method is not limited thereto. For example, the vertical distance 5070 between the imaging / detecting unit 10 and the eye 5011 of the user 5010 can be measured by the method described in the second embodiment, or the user 5010 can directly input the value of the vertical distance 5070.

[0621] Since the calibration processing including the parallax correction mode processing in the present embodiment is basically the same as the processing performed in the second embodiment FIG. 21 , the description thereof is omitted.

[0622] However, in step S3110, the processing described in the second embodiment is added with processing to correct parallax based on the vertical distance 5070( FIG. 33A ) calculated by the parallax correction mode processing of FIG. 32B . That is, the calibration is performed so that the field of view of the user 5010 at infinity coincides with the target field of view 125.

[0623] FIG. 33B is a flowchart showing a subroutine of the recording direction / region determination processing in step S300 of FIG. 7A according to the present embodiment. Hereinafter, the processing will be described by also referring to FIG. 34A , FIG. 34B and FIG. 34C . FIG. 33B The same steps as those of FIG. 7D are denoted by the same reference numerals, and the repeated description is omitted.

[0624] In FIG. 33B , the overall control CPU 101 first obtains distance information relating to the available imaging region (imaging target region) using the distance sensor (distance measuring unit) 5011 (step S5301).

[0625] In the next step S5302, the overall control CPU (creation unit) 101 creates a defocus map 5950( FIG. 34A ; distance map information) based on the distance information obtained in step S5301 (the measurement result using the distance sensor 5100).

[0626] FIG. 34A The defocus map 5950 of FIG. 34CThe illustrated case is created in a case where the observation object 5020 appears indoors. In order to easily show the distance information in the defocus map 5950, six distance regions Al to A6 divided by distance from the imaging / detecting unit 10 are indicated stepwise. The distance region Al is closest to the imaging / detecting unit 10. On the other hand, the defocus map can be created steplessly.

[0627] In the next step S5303, the overall control CPU 101 calculates the direction of the observation object 5020 as seen from the imaging / detecting unit 10 based on the defocus map 5950, the face direction 5015, and the vertical distance 5070( FIG. 32B ). That is, the parallax correction is applied to the observation direction established based on the face direction.

[0628] After that, the processing in steps S301 to S305 of the routine is performed, and the processing exits the subroutine. FIG. 7D

[0629] The use of the defocus map 5950 created in this way and the detection result of the face direction 5015 makes it possible to calculate the direction of the observation object 5020 as seen from the imaging / detecting unit 10. Since there is the parallax described using the FIG. 29A and FIG. 29B , the distance to the observation object 5020 cannot be measured with the distance sensor 5100 unless the defocus map 5950 is created.

[0630] The degree of influence of the parallax described in the present embodiment depends on the distance between the user 5010 and the observation object 5020. When the observation object is far from the user 5010, the influence of the parallax can be ignored. In this case, the image can be extracted by including the target field of view of the observation object, and the image can be recorded by the recording direction / region determination processing in the first embodiment. For example, in a case where the user 5010 observes the observation object 5021( FIG. 34C ) located in the middle distance region A5 a certain distance away from the user 5010, the parallax correction can not be applied to the recording direction in step S5303. This is because the observation object 5021 is also included in the target field of view 5043( FIG. 34B ) established based on the recording direction (observation direction) 5041 estimated based on the face direction 5016 detected by the face direction detecting unit 20.

[0631] On the other hand, the present embodiment can expand the allowable range of the distance between the user 5010 and the observation object of the user 5010 (where the observation object is kept within the target field of view) to a closer side compared to the first embodiment. For example, the user 5010 should observe the observation object 5020 located in the closest region Al a short distance away from the user 5010. In this case, the parallax correction is applied to the recording direction in step S5303. This is because the observation object 5020 is not included in the target field of view 5043 established based on the recording direction (observation direction) 5041 estimated based on the face direction 5016 detected by the face direction detecting unit 20.​FIG. 34A ). In this case, the observation direction (recording direction) 5040 is estimated based on the face direction 5015 detected by the face direction detection unit 20 in the first embodiment. However, the target field of view 5042 established according to this observation direction 5040 does not include the observation object 5020. On the other hand, in the present embodiment, the parallax correction is applied to the observation direction 5040 in step S5303. As a result, the target field of view 5036 including the observation object 5020 is established according to the recording direction after the parallax correction. Therefore, it is also possible to satisfactorily photograph an observation object such as the observation object 5020 whose distance from the user 5010 is short to the extent that the parallax effect cannot be ignored. FIG. 34B ). In this case, the observation direction (recording direction) 5040 is estimated based on the face direction 5015 detected by the face direction detection unit 20 in the first embodiment. However, the target field of view 5042 established according to this observation direction 5040 does not include the observation object 5020. On the other hand, in the present embodiment, the parallax correction is applied to the observation direction 5040 in step S5303. As a result, the target field of view 5036 including the observation object 5020 is established according to the recording direction after the parallax correction. Therefore, it is also possible to satisfactorily photograph an observation object such as the observation object 5020 whose distance from the user 5010 is short to the extent that the parallax effect cannot be ignored. FIG. 33B

[0632] Further, according to the present embodiment, it is possible to record an observation object located in the middle distance region A5 at a position closer to the center of the target field of view. For example, when the user 5010 is observing an observation object 5021 located in the middle distance region A5 (a), if the parallax correction is not applied to the recording direction 5041 as in the first embodiment, a target field of view 5043 in which the observation object 5021 is located at the upper end will be established. On the other hand, in the present embodiment, since the parallax correction is applied to the recording direction 5041 in step S5303, a recording region (target field of view) 5037 in which the observation object 5021 is located at the center is generated according to the recording direction after the parallax correction. FIG. 34A FIG. 33B

[0633] Thus, in the case where the parallax correction of the present embodiment is applied, it is possible to photograph an observation object at a position closer to the center of the extracted image than in the first embodiment.

[0634] In the present embodiment, the parallax correction is performed at the time of calibration so that the field of view of the user at infinity coincides with the target field of view. Then, when an image is photographed, the parallax correction is applied so that the deviation of the recording direction before and after the correction becomes larger as the distance between the user and the observation object becomes shorter. On the other hand, in the calibration processing of the second embodiment, the parallax correction of the present embodiment can be applied to a subject closer to the user than the position of the calibrator 850 or a subject farther from the user than the position of the calibrator 850.

[0635] Next, the sixth embodiment will be described. In the sixth embodiment, the extraction region determination method to be used in the case where the calculation of the observation direction fails will be described using FIG. 35 , FIG. 36A and FIG. 36B .

[0636] ​​​This embodiment is basically explained as a derivative from the first embodiment. Therefore, the same configurations in the camera system of the sixth embodiment as those in the camera system of the first embodiment are denoted by the same reference numerals, and the repeated explanation is omitted. The different configurations will be explained by adding details.

[0637] In the first embodiment, as shown in FIG. 7A In the recording direction / region determination processing of step S300, the target field of view is established based on the viewing direction calculated from the face direction detected by the face direction detection unit 20 in step S200. However, the face direction detection unit 20 can be covered by an obstacle such as a collar and hair, can malfunction, or can be separated from the user. In this case, the face direction of the user cannot be obtained, and the image of the target field of view that the user wants to photograph cannot be captured.

[0638] In JP 2007-74033A, in a case where the second camera for photographing the user cannot detect the user, the detection of the user is retried without storing the failure of the detection in the history of the detection information related to the user. Further, if the detection of the face direction fails during the imaging operation by tracking the face direction, the image that does not greatly deviate from the user's intention is captured by determining the imaging direction according to the situation.

[0639] On the contrary, in this embodiment, in a case where the face direction of the user can be detected, as in the first embodiment, the face direction is detected by the face direction detection unit 20, and the image of the target field of view is captured according to the recording direction calculated based on the viewing direction. On the other hand, in a case where the face direction of the user cannot be detected and the viewing direction of the user cannot be calculated, the image of the target field of view that reflects the user's intention is captured. That is, in this embodiment, after the face direction detection processing of step S200 is completed, the viewing direction determination processing is executed before the recording direction / region determination processing of step S300 is executed. In the viewing direction determination processing, in a case where the face direction detection unit 20 fails in the detection of the face direction of the user, the viewing direction is estimated by determining the user's intention according to the situation. That is, the image of the target field of view in the recording direction based on factors other than the viewing direction calculated from the face direction is captured.

[0640] FIG. 35 is a flowchart of the viewing direction determination processing according to this embodiment, which is executed by the overall control CPU 101. Hereinafter, this processing will be explained by also using FIG. 36A and FIG. 36B .

[0641] In step S6001, it is determined whether or not the face direction is detected by the face direction detection unit 20. In a case where the face direction is obtained, the process proceeds to step S6004. In step S6004, the overall control CPU (mode switching unit) 101 switches the mode of the process to the face direction mode (first imaging mode), and determines the observation direction calculated according to the face direction by the method shown in the first embodiment as the recording direction. After that, the process exits the subroutine.

[0642] On the other hand, in a case where the face direction is not obtained (NO in step S6001), the overall control CPU (mode switching unit) 101 causes the process to proceed to step S6002 to shift to another mode. In step S6002, it is determined whether or not there is any subject that is tracked in the past.

[0643] The determination process in step S6002 will be described using a graph showing the relationship between the observation direction detection state of the user and the captured image for each frame. FIG. 36A

[0644] In the graph in FIG. 36A , "n" denotes the frame number of the image, "θ" denotes the horizontal moving angle of the face of the user, and the user state shows the positional relationship between the user and the observation object in each frame. Further, the entire image shows the super-wide-angle image captured by the imaging unit 40 in each frame, and the captured image shows the image recorded in assistance in each frame and corresponds to the area shown by the dotted line in the entire image.

[0645] As shown in each of the user states in FIG. 36A , the user is observing the subject as the observation object shown by the quadrangle at the bottom position of the screen. FIG. 36A A case where the observation direction of the user cannot be detected in the fifth frame (n = 5) is exemplified.

[0646] In the present embodiment, a time period including four previous frames based on the current frame is defined as a predetermined time period. In a case where the subject that can be determined to be the same three or more times within the predetermined time period is included in the captured image, it is determined that there is a subject that is tracked in the past.

[0647] As shown in FIG. 36A , although the moving angle θ changes by every +10° from the first frame (n = 1) to the fourth frame (n = 4), the subject shown by the quadrangle that can be determined to be the same subject is included in the captured image. Therefore, in the fifth frame (n = 5), it is determined that there is a subject that is tracked in the past. It should be noted that the criterion of the determination in step S6002 can be changed in correspondence with the detection period of the face direction or the accuracy of the face direction detection unit 20.

[0648] Returning to​FIG. 35 In a case where it is determined that there is a same subject that has been tracked in the past predetermined period (YES in step S6002), the processing proceeds to step S6005.

[0649] In step S6005, the mode of the processing is switched to a past subject tracking mode (second imaging mode) in which a past subject direction is determined as a recording direction, and the recording direction is determined to track the past subject. Then, the processing proceeds to step S6008. In this way, in the present embodiment, even if the face direction cannot be detected, since the mode is switched to the past subject tracking mode and the recording direction is determined in a case where there is a subject that has been tracked in the past, the user's immediate intention can be reflected to the image. Since the method of recognizing a subject in a captured image and the subject tracking detection method by the general control CPU (subject recognition unit) 101 are well known, detailed description thereof is omitted.

[0650] On the other hand, in a case where it is determined that there is no subject that has been tracked in the past (NO in step S6002), the processing proceeds to step S6003. In step S6003, it is determined whether a subject that has been pre-registered in the internal nonvolatile memory (subject registration unit) is detected in the latest captured image.

[0651] In the present embodiment, the user specifies an image of a person that the user wants to capture from among the images stored in the display device 800. The display device controller 801 recognizes the features of the person, and pre-registers the subject by transmitting these features to the general control CPU 101 in the camera body 101. Note that the subject detected in step S6003 is not limited thereto. For example, a subject included in a captured image obtained at a read completion timing or another detection timing can be detected in step S6003. Further, a pattern matching technique is used to determine whether the pre-registered subject coincides with the subject in the latest captured image. Since the pattern matching technique is well known, detailed description thereof is omitted.

[0652] When it is determined that the pre-registered subject is detected in the latest captured image (YES in step S6003), the processing proceeds to step S6006. In step S6006, the mode of the processing is switched to a registered subject tracking mode (third imaging mode) in which a registered subject direction is determined as a recording direction, and the recording direction is determined to track the registered subject. Then, the processing proceeds to step S6008.

[0653] On the other hand, when it is determined that the pre-registered subject is not detected in the latest captured image (NO in step S6003), it is determined that the observation object cannot be estimated, and the processing proceeds to step S6007.

[0654] In step S6007, the overall control CPU (field angle changing unit) 101 switches the mode of the process to the subject loss mode (fourth imaging mode) in which the recording direction before the face direction detection failure is maintained and the imaging field angle is wider than the prescribed field angle. Thereafter, the process proceeds to step S6008. Note that the recording direction in the subject loss mode can be continuously moved by the amount of change of the viewing direction before the face direction detection failure.

[0655] Hereinafter, the case where step S6007 enters the subject loss mode will be described using FIG. 36B FIG. 36B An example in which the viewing direction of the user cannot be detected in the fifth frame (n = 5) is illustrated.

[0656] In the example of FIG. 36B , no main subject is found from the first frame (n = 1) to the fourth frame (n = 4), and no pre-registered subject is found in the captured image of the fifth frame (n = 5). Therefore, the viewing direction in the fifth frame (n = 5) is moved to the right by inertia of the movement in the first to fourth frames in the entire image. Further, the field angle extracted from the entire image is changed to a wider angle.

[0657] In step S6008, when the recording direction is determined based on a factor other than the face direction in any of steps S6005 to S6007, the overall control CPU (notification unit) 101 notifies the user of an error showing the face direction detection failure (detection error). Thereafter, the process exits the subroutine. In the present embodiment, the vibrator 106 is used to output a warning to the user. The notification method in step S6008 is not limited to this. Other notification methods can be employed, such as a warning using the LED 17, and display of a warning message on a terminal such as the display device 800 in cooperation with the camera body 1. FIG. 5

[0658] As described above, in the present embodiment, since the recording direction and the field angle are changed according to the situation in which the face direction cannot be detected, the user can avoid the imaging miss of the image of the target field of view that the user originally intended to capture.

[0659] That is, in the present embodiment, when the face direction cannot be detected, and when a subject that was tracked in the past or a pre-registered subject is detected, the subject is tracked. On the other hand, when such a subject cannot be detected, the imaging field angle is made wider than the prescribed field angle to avoid the imaging miss and facilitate the re-detection of the subject.

[0660] Thereby, it is possible to prevent the situation in which an image that the user does not want is captured due to the face direction detection failure.

[0661] ​​Although the processing of steps S6001 to S6008 is performed for each frame, even after transitioning to each mode, the mode can be changed based on mode determination information (such as information regarding whether the facial direction is obtained from the facial direction detection unit 20). For example, in this embodiment, if a pre-registered subject is detected as a result of widening the field angle in the subject loss mode, the mode is transitioned to the registered subject tracking mode in which the direction of the detected subject is determined as the recording direction. In this case, the widened field angle is restored to the specified field angle.

[0662] Furthermore, although the mode is changed by one determination in the present embodiment, the mode may be transitioned based on a plurality of determinations according to the frame rate or the performance of the face direction detection.

[0663] Next, the seventh embodiment will be described. In the seventh embodiment, FIG. 37A to FIG. 40 A method for determining the viewing direction according to the accuracy (reliability) of face direction detection will be described.

[0664] This embodiment will be described essentially as a derivative of the first embodiment. Therefore, configurations of the camera system of the seventh embodiment that are identical to those of the camera system of the first embodiment are denoted by the same reference numerals, and duplicate descriptions are omitted. Different configurations will be described by adding details.

[0665] The sixth embodiment prevents a camera operation in a recording direction that is not intended by the user by switching the mode for determining the viewing direction according to whether the face direction can be detected. On the other hand, when the face direction of the user cannot be stably detected as in JP2007-74033A, an image may be captured at a field angle that is not intended by the user. In the case of FIG. 1B When the imaging / detection unit 10 of the camera body 1 is worn in front of the clavicle as shown, the detection accuracy of the facial direction detection may decrease due to the influence of the collar, hair, etc. If the detection accuracy decreases, the facial direction cannot be stably detected.

[0666] In the case where the user turns his face in the horizontal direction ( FIG. 37B and FIG. 37C ), the area of ​​the jaw and cheek hidden by the body or shoulders is the same as when the user is facing the front direction ( FIG. 37A ) is larger than the area in the camera body 1. In other words, the facial area available for detecting facial directions is narrower in some facial directions. The likelihood of reduced detection accuracy increases in these facial directions. This characteristic is largely dependent on the user's choice of where to wear the camera body 1.

[0667] In the present embodiment, the detection accuracy (reliability) of the face direction is calculated based on the detection result of the wearing position of the camera body 1 and the face direction. In the case where the reliability is high, the face direction is mainly reflected to the observation direction. In the case where the reliability is low, factors other than the face direction are mainly reflected to the observation direction. Thereby, the user's intention can be reflected to the imaging operation.

[0668] FIG. 38 is a flowchart showing the observation direction determination processing at the time of obtaining the face direction according to the present embodiment, which is executed instead of the processing of step S6004. FIG. 35 The processing is executed by the overall control CPU (observation direction determination unit) 101.

[0669] In step S7001, the overall control CPU (first observation direction calculation unit and reliability calculation unit) calculates the face direction reliability T n based on the face direction (first observation direction) θ n obtained by the face direction detection unit 20 at the time of imaging the image of the frame n.

[0670] The face direction reliability T n is calculated as follows. First, the face direction θ n is divided into three components of the face direction θ yaw , the face direction θ pitch , and the face direction θ roll . The face direction θ yaw is a rotation component of the face movement in the horizontal direction. The face direction θ pitch is a rotation component of the face movement in the vertical direction. The face direction θ roll is a rotation component of the tilt of the head.

[0671] Since the present embodiment assumes that the user wears the camera body on the user's collarbone and detects the face direction from a position below the face, the face direction reliability T n (0≤T n ≤1) is found by the following formula 701.

[0672]

[0673] FIG. 39 The relationship between the face direction θ yaw and the face direction reliability T n is shown. FIG. 39 The graph in FIG. 7 shows that the face direction reliability T yaw becomes lower as the angle of the face direction θ n becomes larger with respect to the front.

[0674] In the present embodiment, the face direction reliability T is calculated using the formula 701 n On the other hand, the face direction reliability can be obtained by weighted average values calculated by weighting the past face direction reliabilities according to the detection accuracy of the face direction by the face direction detection unit 20 and the frame frequency of detection. Further, in calculating the face direction reliability T n , the accuracy of pattern matching, the wearing position, and the like can be weighted.

[0675] Further, in the present embodiment, the face direction reliability of the estimated viewing direction is calculated by the formula 701. However, the method of calculating the face direction reliability is not limited thereto. For example, a face direction reliability adjusted according to the wearing point of the camera body 1 estimated by the calibration of the second embodiment can be adopted. Further, in a case where it is judged that the detection accuracy is low at the time of calibration, the face direction reliability can be changed according to the detection accuracy. Further, in a case where the face direction is detected using machine learning, the precision ratio can be reflected to the face direction reliability.

[0676] In step S7002, the overall control CPU 101 calculates the angular velocity ω n of the movement of the face. Specifically, the angular velocity ω n is calculated using the face direction θ n obtained by the face direction detection unit 20 at the time of imaging of the image of the frame n, the face direction obtaining time t n-1 , the face direction θ n-1 of the previous frame stored in the main memory 103, and the face direction obtaining time t n of the previous frame by the following formula 702.

[0677]

[0678] Although the angular velocity ω n is calculated using the information relating to the current frame and the information relating to the previous frame in the present embodiment, one or more past information can be used to calculate the angular velocity according to the frame frequency and the like.

[0679] In step S7003, the overall control CPU (viewing direction estimation unit) 101 estimates the current face direction according to the transition of the past face direction stored in the main memory 103. In the present embodiment, a time period including four previous frames based on the current frame is defined as a predetermined time period. When the continuous change of the face direction in a certain direction is judged three or more times within the predetermined time period, it is judged that the viewing direction can be estimated from the past face direction and the angular velocity. Further, in the estimation, the estimated angular velocity ωave and the estimated face direction θ is calculated by the following equation 704 ave (first observation direction). The calculations of the equations 703 and 704 respectively correspond to the processes al and a2 shown in FIG. 6. FIG. 40

[0680] It should be noted that the length of the predetermined time period and the weight of the weighted average used in step S7003 can be changed according to the frame frequency and the detection accuracy of the face direction detection unit 20.

[0681]

[0682] θ ave = θ n-1 + (t n - t n-1 ) · ω ave ... Equation 704

[0683] In step S7004, the overall control CPU 101 estimates the observation direction using internal information other than the information from the face direction detection unit 20 among the information stored in the main memory 103. Specifically, in the present embodiment, it is judged based on the subject detection history whether or not the subject is currently being tracked. When it is judged that the subject is currently being tracked, the estimated observation direction θ sub (second observation direction) is calculated based on the movement of the subject. In the present embodiment, a time period including four previous frames based on the current frame is defined as the predetermined time period. When a subject that can be judged as the same subject is detected three or more times within the predetermined time period, it is judged that the subject is currently being tracked. The criteria of the subject tracking judgment can be changed corresponding to the period and accuracy of the detection with the overall control CPU 101. Since the subject tracking detection technique is well known, the detailed explanation thereof is omitted.

[0684] Although the internal information used for the estimation of the observation direction in step S7004 is the subject detection history in the present embodiment, it is not limited thereto. For example, the face information of the user photographed by the imaging unit 40 or the information related to the movement and posture of the camera body 1 detected by the angular velocity sensor 107 and the acceleration sensor 108 can be used to estimate the observation direction according to the wearing position and performance of the camera body 1. Further, in the case where there is a pre-registered subject, as in step S6006 in the sixth embodiment, the overall control CPU (third observation direction estimation unit) 101 can determine the direction of the pre-registered subject in the latest photographed image as the estimated observation direction θ sub .

[0685] ​In step S7005, the overall control CPU 101 stores the face direction detection related information as history in the main memory 103. The face direction detection related information includes the angular velocity ω of the movement of the face generated in step S7002. n , the face direction reliability T calculated in step S7001 n , the facial direction θ detected by the facial direction detection unit 20 n , facial direction acquisition time t n and the time points at which these values ​​were generated.

[0686] In step S7006, the overall control CPU 101 determines whether the face direction reliability T calculated in step S7001 is n Is it equal to or greater than a predetermined value? n If the face direction is equal to or greater than a predetermined value, it is determined that the face direction reliability is high, and the process proceeds to step S7009. In step S7009, the overall control CPU 101 determines the face direction as the current observation direction θ′. n . Then, the processing enters step S7013.

[0687] On the other hand, when the face direction reliability T calculated in step S7001 is n If it is less than the predetermined value (No in step S7006), the process proceeds to step S7007. In step S7007, it is determined whether the estimated face direction θ can be calculated in step S7003. ave , and |θ n -θ ave | Is it equal to or less than a predetermined angle? When both conditions are met, the process proceeds to step S7010. In this embodiment, the predetermined angle is set to π / 8 in this determination.

[0688] In step S7010, the overall control CPU (first viewing direction estimation unit) 101 uses the face direction θ n , estimate the observation angle θ ave and facial direction reliability T n To determine the current viewing direction θ' n In this embodiment, the current viewing direction θ' is calculated by the following formula 705: n , and the process proceeds to step S7013. The calculation of formula 705 is the same as FIG. 40 The processing b1 shown in FIG. FIG. 39 As shown, as the facial angle θ yaw The absolute value of the face direction reliability T becomes smaller. n Therefore, at the facial angle θ yawWhen the absolute value of is small, as shown in formula 705, the facial direction θ n To a large extent, it reflects the current observation direction θ' n On the other hand, at the facial angle θ yaw When the absolute value of is large, as shown in formula 705, except for the facial direction θ n Factors other than the estimated face direction θ' ave ) is largely reflected in the current observation direction θ′ n .

[0689] θ′ n =T n ·θ n +(1-T n )·θ ave ...Formula 705

[0690] If the above conditions are not met in step S7007, the process proceeds to step S7008. In step S7008, it is determined whether the estimated viewing direction θ can be calculated. sub , and |θ n -θ sub | Is it equal to or less than a predetermined angle? If the condition in step S7008 is met, the process proceeds to step S7011. In this embodiment, similar to step S7010, the predetermined angle is set to π / 8 in the judgment.

[0691] In step S7011, the overall control CPU (second viewing direction estimation unit) 101 uses the face direction θ n , estimate the viewing direction θ sub and facial direction reliability T n To determine the current viewing direction θ′ n In this embodiment, the current viewing direction θ′ is obtained by the following formula 706: n , and the process proceeds to step S7013. FIG. 39 As shown, in the same manner as step S7010, as the facial angle θ yaw The absolute value of the face direction reliability T becomes smaller. n Therefore, at the facial angle θ yaw When the absolute value of is small, as shown in formula 706, the facial direction θn is largely reflected in the current observation direction θ′ n On the other hand, at the facial angle θ yaw When the absolute value of is large, except for the face direction θ n Factors other than the estimated observation direction θ sub ) is largely reflected in the current observation direction θ′ n .

[0692] θ′ n =T n ·θ n +(1-T n )·θ sub ...Formula 706

[0693] If the above conditions are not satisfied in step S7008, it is determined that a reliable viewing direction cannot be obtained in the current situation, and the process proceeds to step S7012. In S7012, by making the previous viewing direction θ′ n-1 Determine the current viewing direction θ′ based on the past viewing direction changes by inertial movement n , and the field angle is made wider than the prescribed field angle. The process then proceeds to step S7013. This reduces the possibility that the user will miss photographing the subject that the user wants.

[0694] Although in this embodiment, the face direction reliability T n and the detection state of the subject to switch the current observation direction θ′ n For example, when calculating the estimated face direction θ ave and estimated viewing direction θ sub In this case, the calculated observation direction θ′ can be corrected according to the calculated reliability. n .

[0695] Furthermore, if the calculated reliability is not greater than a predetermined value, the user is more likely to miss capturing the intended subject. Therefore, it is preferable to widen the angle of field relative to the prescribed angle of field. In this case, the process may proceed to step S7012. Subsequently, if one of the calculated reliability levels becomes greater than a predetermined value, the widened angle of field is preferably restored to the prescribed angle of field.

[0696] As FIG. 38 As a result of the processing, the face direction reliability T n When the face direction θ is high, n Determined as the current viewing direction θ' n On the other hand, in the face direction reliability T n If the face direction reliability is low, use the face direction reliability T n The current viewing direction θ' is determined based on the facial direction obtained under the n (Recording direction) In addition, if necessary, the field angle is widened.

[0697] That is, due to the face direction reliability T nIn a case where the detection accuracy of the estimated face direction is low ave or the estimated observation direction θ sub is low. Thus, it is possible to prevent a situation where an image that the user does not want is captured due to a failure in face direction detection.

[0698] Next, the eighth embodiment will be described. In the eighth embodiment, a method of wearing the camera body 1 in a stable position will be described using FIG. 41A to FIG. 45G . Basically, the present embodiment will be described as a derivative from the first embodiment. Therefore, the same configurations in the camera system of the eighth embodiment as those in the camera system in the first embodiment are denoted by the same reference numerals, and the repeated description is omitted. The different configurations will be described by adding details.

[0699] First, the angle adjustment of the connection members 80L and 80R (neck suspension members) will be described. FIG. 41A 、 FIG. 41B and FIG. 41C are enlarged side views of the imaging / detecting unit 10. Although the following description exemplifies the left connection member 80L, the right connection member 80R is similarly adjusted.

[0700] FIG. 41A is a view showing a state where the connection member 80L is set in the standard position Ax0. FIG. 41B is a view showing a state where the connection member 80L is rotated by an angle θA1 about the rotation axis OA with respect to the standard position Ax0. FIG. 41C is a schematic view showing a mechanical structure inside the angle holding member 81L that can be seen when the outside of the angle holding member 81L is removed.

[0701] As shown in FIG. 41C , an angle adjustment mechanism (neck suspension angle adjustment mechanism) 8100 is arranged inside the angle holding member 81L.

[0702] The angle adjustment mechanism 8100 is composed of an angle adjustment cam 8101 that adjusts the angle of the angle holding member 81L with respect to the imaging / detecting unit 10 and a latch member 8102 that latches the angle adjustment cam 8101. It should be noted that the rotation axis OA of the angle holding member 81L coincides with the center of the angle adjustment cam 8101.

[0703] The latch member 8102 applies a force to the angle adjustment cam 8101 with a spring (not shown). In pressing the angle adjustment button 85 FIG. 2F) and the latching member 8102 can be separated from the angle adjustment cam 8101. That is, only during the pressing of the angle adjustment button 85L, the angle holding member 81L of the connection member 80L becomes rotatable with respect to the imaging / detecting unit 10.

[0704] In a case where the angle holding member 81L is rotated with respect to the imaging / detecting unit 10 during the pressing of the angle adjustment button 85L, the user can adjust the connection member 80L from the standard position Ax0 FIG. 41A ) to the position Ax1 FIG. 41B ).

[0705] Although the present embodiment adopts the stepped adjustment mechanism composed of the angle adjustment cam 8101 and the latching member 8102 as the mechanism for holding the angle of the angle holding member 81L with respect to the imaging / detecting unit 10, a stepless adjustment mechanism using sliding resistance can be adopted.

[0706] Further, although the present embodiment adopts the structure in which the user rotates the angle holding member 81L during the pressing of the angle adjustment button 85L, it is not limited thereto. For example, a structure that does not require the angle adjustment button 85L can be used. Such a structure enables the rotation of the angle holding member 81L when an external force larger than a threshold is applied. For example, a ball can be used instead of the latching member 8102, and sliding resistance can be used.

[0707] FIG. 42A 、 FIG. 42B and FIG. 42C are side views showing a state in which a user wears the camera body 1. FIG. 42A is a view showing a user wearing the camera body 1 in which the connection member 80L is disposed at the standard position Ax0 and the belt portion 82L is elongated. FIG. 42B is a view showing a user wearing the camera body 1 in which the connection member 80L is disposed at the standard position Ax0 and the belt portion 82L is shortened. FIG. 42C is a view showing a user wearing the camera body 1 in which the connection member 80L is disposed at the position Ax1 and the belt portion 82L is shortened.

[0708] As shown in FIG. 42A and FIG. 42C , in a case where the relationship between the position of the connection member 80L and the length of the belt portion 82L is suitable for the user, the imaging lens 16 is directed to the front of the user. On the other hand, as shown in FIG. 42B , in a case where the relationship between the position of the connection member 80L and the length of the belt portion 82L is not suitable for the user, the imaging lens 16 is not directed to the front of the user. In FIG. 42B , the optical axis of the imaging lens 16B is upward.

[0709] Thus, since the connecting member 80L is configured so that its position is adjustable, the user can wear the camera body 1 so that the optical axis of the imaging lens 16 will be approximately parallel to the line of sight in the natural state of the user. In the case where the optical axis of the imaging lens 16 matches the horizontal direction when the user wears the camera body 1 in the appropriate position, similarly, appropriate wear is available.

[0710] Next, adjustment of the angle of the chest contact pads 18a and 18b will be explained. FIG. 43A FIG. 43B and FIG. 43C is an enlarged side view showing the imaging / detecting unit 10 without showing the connecting members 80L and 80R. Although the following explanation exemplifies the left chest contact pad 18a, the right chest contact pad 18b is similarly adjusted.

[0711] FIG. 43A is a view showing a state where the chest contact pad 18a is set in the standard position BxO. FIG. 43B is a view showing a state where the chest contact pad 18a is rotated by an angle ΘB1 from the standard position BxO about the rotation axis OB. FIG. 43C is a schematic view showing the mechanical structure inside the imaging / detecting unit 10 which can be seen in the case where the housing of the imaging / detecting unit 10 is removed.

[0712] As shown in FIG. 43C , a contact angle adjustment mechanism 8200 is arranged inside the imaging / detecting unit 10. The contact angle adjustment mechanism 8200 is composed of an angle adjustment cam 8201 which adjusts the angle of the chest contact pad 18a with respect to the imaging / detecting unit 10, and a latch member 8202 which latches the angle adjustment cam 8201. FIG. 43A to FIG. 43C The rotation axis OB shown is the center of rotation of the chest contact pad 18a.

[0713] The latch member 8202 is urged to the angle adjustment cam 8201 by a spring (not shown). During depression of the angle adjustment button 8203, the urging is released and the latch member 8202 can be separated from the angle adjustment cam 8101. That is, only during depression of the angle adjustment button 8203, the chest contact pad 18a becomes rotatable with respect to the imaging / detecting unit 10.

[0714] In the case where the chest contact pad 18a is rotated with respect to the imaging / detecting unit 10 during depression of the angle adjustment button 8203, the user can adjust the chest contact pad 18a from the standard position BxO to the position Bx1.

[0715] ​Although the present embodiment adopts the stepped adjustment mechanism composed of the angle adjustment cam 8201 and the latch member 8202 as the mechanism for holding the angle of the chest contact pad 18a with respect to the imaging / detecting unit 10, a stepless adjustment mechanism using sliding resistance can be adopted.

[0716] Further, although the present embodiment adopts the structure in which the user turns the chest contact pad 18a during pressing of the angle adjustment button 8203, it is not limited thereto. For example, a structure that does not require the angle adjustment button 8203 can be used. Such a structure enables turning of the chest contact pad 18a when an external force greater than a threshold is applied. For example, a ball can be used instead of the latch member 8202, and sliding resistance can be used.

[0717] FIG. 44A 、 FIG. 44B and FIG. 44C are side views showing the state in which the user wears the camera body 1 without showing the connection members 80L and 80R. FIG. 44A shows the state in which the user with a standing chest wears the camera body 1 with the chest contact pad 18a disposed at the standard position BxO. FIG. 44B shows the state in which the user with a flat chest wears the camera body 1 with the chest contact pad 18a disposed at the standard position BxO. FIG. 44C shows the state in which the user with a flat chest wears the camera body 1 with the chest contact pad 18a disposed at the position Bx1.

[0718] As shown in FIG. 44A and FIG. 44C , in the state in which the position of the chest contact pad 18a is suitable for the inclination of the user's chest, the chest contact pad 18a contacts the user's chest in a wide area. On the other hand, as shown in FIG. 44B , in the state in which the position of the chest contact pad 18a is not suitable for the inclination of the user's chest, the chest contact pad 18a contacts the user's chest in only a few areas. In the case where the area in which the chest contact pad 18a contacts the user's chest becomes small as shown in FIG. 44B , the imaging / detecting unit 10 will easily deviate from the user's body due to the movement of the user's body, which results in a large blur in the captured image.

[0719] Since the chest contact pad 18a is configured to be easily adjusted in angle, the user can wear the camera body 1 so that the chest contact pad 18a contacts the user's chest in a wide area, which reduces the blur in the captured image.

[0720] In the present embodiment, although the chest contact pad 18a is disposed in the imaging / detecting unit 10, the chest contact pad 18a can be disposed in the connection member 80L. Even in this case, a similar effect is obtained. In this case, for example, the connection member 80L can be used instead of the chest contact pad 18a. FIG. 41CA similar mechanism to the illustrated angle adjustment mechanism 8100 is arranged inside the connection member 80L as a mechanism for adjusting the angle of the chest contact pad 18a with respect to the connection member 80L.

[0721] Next, the structure of the band portion 82L and the cable 84 will be described in detail. As described in the first embodiment, the battery unit (power supply unit) 90 of the camera body 1 and the imaging / detecting unit 10 are separate modules that are electrically connected by the cable 84.

[0722] If the cable 84 and the band portion 82L are separated, the appearance of the camera body 1 is poor, and the wearing operation of the camera body 1 becomes troublesome. This is not preferable. Therefore, it is preferable to integrate the band portion 82L and the cable 84. On the other hand, the integrated structure is not limited to FIG. 2B the illustrated structure.

[0723] FIG. 45A to FIG. 45G is a diagram showing various structures of the band portion 82L and the connecting surface 83L that is a cross section of the cable 84 integrated with the band portion 82L. Figures 45A-45C shows a structure in which the cable 84 is composed of a flexible substrate (FPC). Figures 45D-45G shows a structure in which the cable 84 is composed of a thin cable.

[0724] Figure 45A and Figure 45D shows a structure in which the cable 84 is embedded inside the band portion 82L as viewed from the connecting surface 83L. In this case, the band portion 82L is preferably made of an elastic material (such as silicone rubber, elastomer, rubber, and plastic) that enables injection molding. For example, the band portion 82L is inserted into the cable 84 at the time of injection molding. In addition to this, the band portion 82L can be composed of two components. In this case, the cable 84 is sandwiched between the components of the band portion 82L, and they are integrated by an adhesive or heat welding. The manufacturing method is not limited to the above two methods. Any other method can be employed as long as the band portion 82L and the cable 84 are integrated as shown in Figure 45A and Figure 45D .

[0725] Figure 45B , Figure 45C and Figure 45E shows a structure in which the cable 84 is connected to the outside of the band portion 82L as viewed from the connecting surface 83L. Figure 45B shows a structure in which the cable 84 is attached to the band portion 82L. The band portion 82L does not have a specific structure that is integrated with the cable 84. This structure can be manufactured at low cost. When the cable (FPC in this case) 84 appears on the outside, the appearance of the product can be improved by coating the FPC or by covering the FPC with a film. Furthermore, in Figure 45BWhen the cable (FPC in this case) 84 is arranged on the inner side (neck portion) of the band portion 82L in the illustrated structure, the wearing feeling can be improved by coating the FPC or by covering the FPC with a film.

[0726] Figure 45C and Figure 45E A structure in which a recess 83a is formed in the band portion 82L to be integrated with the cable 84 is shown as viewed from the connecting surface 83L. The cable 84 is disposed in the recess 83a. In this case, when the recess 83a is arranged on the inner side (neck portion side) of the band portion 82L, a good appearance can be maintained, and since the cable 84 is stored in the recess 83a and does not directly contact the user's neck, a good wearing feeling is also maintained without special treatment. Furthermore, since the recess 83a does not require additional costs if designed before manufacturing, there is an advantage in terms of cost.

[0727] Figure 45F and Figure 45G A structure in which the cable 84 is embedded inside the band portion 82L is shown as viewed from the connecting surface 83L. Figure 45F A structure in which the cable 84 consists of a single wire is shown. Figure 45G A structure in which the cable 84 consists of three wires is shown. Figure 45F and Figure 45G The characteristic feature of the structure in Figure 45A and Figure 45D is that the cross-sectional area in the connecting surface 83L of the band portion 82L is ensured. This is different from the structures in

[0728] The protruding side of the cable 84 is preferably arranged on the outer side of the band portion 82L to obtain a comfortable wearing feeling. Figure 45F and Figure 45G The structure in

[0729] As described above, if the balance between the appearance and the wearing feeling is prioritized, the structure in Figure 45C or Figure 45E has an advantage. If the cost or the rigidity is prioritized, another structure in Figure 45A , Figure 45B , Figure 45D , Figure 45F or Figure 45G may be employed.

[0730] Next, the ninth embodiment will be described. In the ninth embodiment, a modification of the camera system including the camera body 1 will be described using Figure 46A and Figure 46B . Basically, the present embodiment will be described as a derivation from the first embodiment. Therefore, the configurations in the camera system of the ninth embodiment that are the same as the configurations in the camera system in the first embodiment are denoted by the same reference numerals, and the repeated description is omitted. The different configurations will be described by adding details.

[0731] The display device 800 in the first embodiment uses a general smart phone. There are various smart phones in commercial scenes, and their arithmetic capabilities are also different from each other. For example, the display device 800 in the first embodiment has a relatively high arithmetic capability. Therefore, when the camera body 1 transmits the image of the recording direction extracted from the super-wide-angle image to the display device 800, information required for the optical correction processing or the image stabilization processing is added to the image. The display device 800 in the first embodiment performs the distortion correction processing and the image stabilization processing based on the added information. However, for a smart phone having a relatively low arithmetic capability, such processing is difficult.

[0732] The camera system of the present embodiment is provided with the camera body 1' including an image pickup device, and the display device 9800 having an arithmetic capability lower than that of the display device 800. When the camera body 1' performs the processing from the preparation processing to the main recording processing (steps S100 to S600 in Figure 7A , the camera body 1' performs the optical correction processing and the image stabilization processing (steps S800 and S900) without performing the transmission processing to the display device (step S700). After that, the camera body 1' performs the processing for transmitting the image subjected to the processing in steps S800 and S900 to the display device 9800.

[0733] On the other hand, the display device 9800 performs the auxiliary recording processing (S1000) on the image from the camera body 1' without performing the processing of steps S800 and S900.

[0734] The camera system of the present embodiment will be specifically described below. Figure 46A is a block diagram showing a hardware structure of the display device 9800 connected to the camera body 1' including the image pickup device according to the present embodiment.

[0735] In Figure 46A , the hardware structure of the display device 9800 that is the same as the hardware structure of the display device 800 according to the first embodiment shown in Figure 6 is denoted by the same reference numerals, and the repeated description is omitted.

[0736] The display device 9800 has a display device controller 9801 instead of the display device controller 801 of the display device 800 , and does not have the face sensor 806 .

[0737] The display device controller 9801 is composed of a display device controller 801 ( Figure 6 ) of the CPU. In addition, the capabilities of the internal non-volatile memory 812 and the main memory 813 can be lower than those in the first embodiment.

[0738] Figure 46B 1 ' is a functional block diagram showing the camera body 1 '. Figure 46B In the embodiment, the camera body 1' and Figure 4 The same functional blocks as those of the illustrated hardware blocks of the camera body 1 according to the first embodiment are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0739] Figure 46B Functional block diagram and Figure 4 The functional block diagram of FIG. 1 differs from FIG. 2 in the following points. Specifically, an optical correction / image stabilization unit 9080 is provided, which performs optical correction processing and image stabilization processing. Furthermore, an overall control CPU 9101 is provided in place of overall control CPU 101. Furthermore, transmission unit 70 communicates with display device 9800 rather than display device 800.

[0740] That is, in this embodiment, the optical correction / image stabilization unit 9080 of the overall control CPU 9101 performs optical distortion correction and image stabilization processing using the optical correction value and the gyro data. Therefore, since the transmission unit 70 transmits the video file to the display device 9800 after applying the optical correction processing and the image stabilization processing in this embodiment, the data size of the video file in this embodiment is smaller than the data size of the video file 1000 transmitted by the transmission unit 70 to the display device 800 in the first embodiment.

[0741] Furthermore, since the display device 9800 does not perform the processing of steps S800 and S900, the display device 9800 does not require high computing power equivalent to that of the display device 800. Furthermore, images captured by the camera body 1' can be viewed on a simplified display device (viewing device) 900 such as a smartwatch.

[0742] Next, the tenth embodiment will be described. In the tenth embodiment, Figure 47 and Figure 48A modification of the camera system including the camera body 1 will be described. Basically, the present embodiment will be described as a derivative from the first embodiment. Therefore, the configurations in the camera system of the tenth embodiment that are the same as those of the camera system in the first embodiment are denoted by the same reference numerals, and the repeated description will be omitted. The different configurations will be described by adding details.

[0743] In the ninth embodiment, instead of using the display device 9800 having low computational ability, the camera body 1' needs high performance. In the case where the performance of the camera body is improved, the cost of the overall control CPU and its peripheral devices can increase, and heat generation due to processing load can occur. Therefore, in the tenth embodiment, a structure that reduces the computational ability of the camera body and increases the computational ability of the display device will be described.

[0744] Figure 47 is a functional block diagram showing the camera system of the present embodiment including the camera body 1001 and the display device 1080. In Figure 47 , the functional blocks of the camera body 1001 that are the same as the hardware blocks of the camera body 1 according to the first embodiment shown in Figure 4 or the camera body 1' according to the ninth embodiment shown in Figure 46B are denoted by the same reference numerals, and the repeated description will be omitted.

[0745] Figure 47 The functional block diagram shown in Figure 4 and Figure 46B is significantly different from that of the display device 800 in that the display device 1080 is provided with a recording direction / field angle determination unit 1083, an image extraction / visualization unit 1084 for extracting and visualizing images, and an optical correction / image stabilization unit 1085 for performing optical correction processing and image stabilization processing.

[0746] The camera body 1001 is added with a face image main processing unit 1030 for processing the face image detected by the face direction detection unit 20, a main image main processing unit 1050 for processing the main image captured by the imaging unit 40, and an image combination unit 1055 for combining these images. The recording direction / field angle determination unit 1083 and the image extracti...

Claims

1. A wearable camera device comprising: a viewing direction detection unit that is worn around the user's neck and is configured to detect the user's viewing direction based on an image of the user's chin captured by the imager; a camera unit, which is worn on the user's body and is configured to capture images; as well as an image output unit configured to output an image corresponding to the observation direction based on the image captured by the imaging unit, Wherein, the viewing direction detection unit includes: an infrared irradiation unit configured to irradiate an infrared irradiation surface including a user's chin with infrared light; and an infrared detection unit configured to detect reflected light of the infrared light reflected by the infrared irradiation surface using the imager, wherein the viewing direction detection unit outputs the user's viewing direction in the horizontal direction as an angle in a first detection direction, and outputs the user's viewing direction in the vertical direction as an angle in a second detection direction perpendicular to the first detection direction, and The observation direction detection unit calculates the angle in the second detection direction based on the intensity of the reflected light at the chin of the user.

2. The wearable camera device according to claim 1, wherein The viewing direction detection unit detects a user's viewing direction as a three-dimensional viewing direction.

3. The wearable camera device according to claim 1, wherein The second detection direction ranges from -60° to +50°.

4. The wearable camera device according to claim 1, wherein The observation direction detection unit obtains distance information about each distance area of ​​the infrared irradiation surface from the reflected light of the infrared light detected by the infrared detection unit, and detects the observation direction based on the distance information.

5. The wearable camera device according to claim 4, wherein: The viewing direction detection unit detects a rotation center of the user's head and the chin position based on the distance information, and detects the viewing direction according to the rotation center of the head and the chin position.

6. The wearable camera device according to claim 5, wherein: The viewing direction detection unit sets the head rotation center at a position that is closest to the infrared detection unit and is located at the center in the lateral direction of a distance area in which the relative distance to the infrared detection unit is shortest among the distance areas.

7. The wearable camera device according to claim 5, wherein: The viewing direction detection unit sets the chin position at a position that is farthest from the head rotation center and close to a distance region in which a rate of change of the distance from the infrared detection unit is greatest among the distance regions.

8. The wearable camera device according to claim 7, wherein: The viewing direction detection unit calculates a movement angle of the chin position around the head rotation center as an angle in the first detection direction.

9. The wearable camera device according to claim 1, wherein: The image output unit extracts an image corresponding to the observation direction from the image captured by the imaging unit and outputs the extracted image.

10. The wearable camera device according to claim 1 , further comprising a calibration unit configured to calibrate the viewing direction detection unit using a calibrator that can be wirelessly connected to the wearable camera device. in, The calibrator is provided with a face detection unit for projecting infrared light and detecting a user's face, and The observation direction detection unit does not detect the observation direction during a period in which the face detection unit is projecting the infrared light.

11. The wearable imaging device according to claim 1 , further comprising a metadata generation unit configured to generate metadata including in-image position information indicating a position and a size of an image corresponding to the viewing direction relative to an image of each frame output to the image output unit. in, The image output unit generates a video file in which the metadata and the image of each frame are encoded for each frame.

12. The wearable imaging device according to claim 11 , further comprising an optical correction value obtaining unit configured to obtain an optical correction value corresponding to an optical design of an imaging lens in the imaging unit. in, The optical correction values ​​are included in the metadata.

13. The wearable imaging device according to claim 11 , further comprising a movement amount detection unit configured to detect movement of the wearable imaging device and obtain a movement amount, in, The movement amount is included in the metadata.

14. The wearable camera device according to claim 13, wherein: The movement amount detection unit is one of an acceleration sensor for detecting acceleration, an angular velocity sensor for measuring angular velocity, and a magnetometry sensor for measuring the direction of a magnetic field.

15. The wearable camera device according to claim 1, wherein The viewing direction detection unit includes a facial direction detection unit, wherein the facial direction detection unit is configured to detect the facial direction of the user. The wearable camera device further includes: an acquisition and detection unit configured to acquire, during calibration, an image including the positioning index from the camera unit for photographing the positioning index, and detect the facial direction using the facial direction detection unit; a position calculation unit configured to calculate the position of the positioning indicator in the image captured during the calibration based on the shape of the positioning indicator included in the image obtained by the obtaining and detecting unit; a generating unit configured to generate information showing a relationship between the face direction detected by the obtaining and detecting unit and the position of the positioning index calculated by the position calculating unit; and A first calibration unit is configured to calibrate a center position of a target field of view corresponding to the facial direction detected by the facial direction detection unit based on the information generated by the generation unit.

16. The wearable camera device according to claim 1, wherein: The viewing direction detection unit includes a facial direction detection unit, wherein the facial direction detection unit is configured to detect the facial direction of the user. The wearable camera device further includes: a calculation unit configured to calculate an angular velocity of the user's face based on the facial direction detected by the facial direction detection unit; and a recording direction determining unit configured to determine a recording direction of the image output from the image output unit based on the observation direction, In which, when it is determined as a calculation result of the calculation unit that the user's face moves at an angular velocity exceeding a predetermined angular velocity for more than a first predetermined time period, the recording direction determination unit changes the recording direction to a delayed direction with respect to the movement delay of the detected face direction.

17. The wearable camera device according to claim 16, wherein: Even when the calculation result of the calculation unit determines that the user's face moves at an angular velocity exceeding the predetermined angular velocity for more than the first predetermined time period, the image output unit outputs the entire image captured by the camera unit during the camera operation of the moving subject.

18. The wearable camera device according to claim 17, wherein: The recording direction determination unit stops changing the recording direction to the delay direction in a case where a time period that has elapsed after the recording direction is changed to the delay direction exceeds a second predetermined time period.

19. The wearable camera device according to claim 16, wherein: The recording direction determination unit does not change the recording direction when it is determined as a calculation result of the calculation unit that the time period during which the user's face moves at an angular velocity exceeding a predetermined angular velocity is less than the first predetermined time period.

20. The wearable camera device according to claim 16, wherein In a case where the image output from the image output unit is switched from the image of the delay direction to the image corresponding to the viewing direction detected by the viewing direction detection unit, an image effect is added.

21. The wearable camera device according to claim 1, further comprising: a movement amount detection unit configured to detect an amount of movement of the wearable camera device during a video image capturing operation of the camera unit; as well as an observation direction correction unit configured to correct the amount of movement in the observation direction, wherein, when the movement amount detection unit detects that the wearable imaging device is accelerating, the viewing direction correction unit delays the movement amount in the viewing direction; and Here, when the movement amount detection unit detects that the wearable imaging device is decelerating, the viewing direction correction unit accelerates the movement amount in the viewing direction to cover the delay amount.

22. The wearable camera device according to claim 21, wherein: The movement amount detection unit detects the movement amount by comparing images of a current frame and a previous frame obtained by the imaging unit during a video image capturing operation.

23. The wearable camera device according to claim 21, wherein The image output unit outputs a portion of the image extracted according to the observation direction.

24. The wearable camera device according to claim 21, further comprising: A driving mechanism, used for driving the camera unit in a yaw direction and a pitch direction; as well as An imaging direction changing unit is configured to control the driving mechanism to change the imaging direction of the imaging unit based on the observation direction.

25. The wearable camera device according to claim 21, wherein The viewing direction correction unit corrects the amount of movement in the viewing direction so that a speed of movement in the viewing direction will become constant in the video image output from the image output unit.

26. The wearable imaging device according to claim 1, wherein The viewing direction detection unit and the imaging unit are integrally formed. The wearable camera device further includes: a distance measuring unit configured to measure the distance from the imaging unit to an imaging target area; a creating unit configured to create distance map information related to the imaging target area according to the measurement result of the distance measuring unit; and A calculation unit is configured to calculate a direction of an observation object of the user as seen from the imaging unit based on the observation direction, the distance map information, and a vertical distance between the imaging unit and an eye position of the user.

27. The wearable camera device according to claim 26, further comprising: a posture detection unit configured to detect a horizontal axis of the camera unit; an angle calculation unit configured to calculate an angle formed between the detected horizontal axis and a direction of an external positioning indicator seen from the camera unit; A vertical distance calculation unit is configured to calculate the vertical distance based on the angle calculated by the angle calculation unit and the distance between the camera unit and the positioning index measured by the distance measurement unit.

28. The wearable camera device according to claim 27, wherein: The observation direction detected by the observation direction detection unit is calibrated based on the calculated vertical distance, the observation direction detected by the observation direction detection unit when the positioning indicator is located at each specified position, and the distance between the camera unit and the positioning indicator measured by the measuring unit.

29. The wearable camera device according to claim 16, further comprising: a second recording direction determining unit configured to determine a direction determined based on a factor other than the observation direction as the recording direction; as well as Mode switching unit, used to switch camera modes, wherein, when the facial direction detection unit is able to detect the facial direction during the video image capturing operation, the mode switching unit switches the immediately preceding mode to a first camera mode in which the recording direction is determined by the recording direction determination unit during the video image capturing operation, and Wherein, when the facial direction detection unit cannot detect the facial direction, the mode switching unit switches the previous mode to one of the other camera modes, in which the second recording direction determination unit is used to determine the recording direction during the video image shooting operation.

30. The wearable camera device according to claim 29, further comprising a subject recognition unit configured to recognize a subject from an image in the recording direction of a frame of a video image captured by the camera unit, in, When the observation direction detection unit cannot detect the observation direction and the subject recognition unit recognizes the same subject within the past predetermined time period, the mode switching unit sets factors other than the observation direction as the direction of tracking the same subject, and switches the immediate preceding mode to a second camera mode which is one of the other camera modes.

31. The wearable imaging device according to claim 30, further comprising a subject registration unit configured to pre-register a subject to be detected. in, In the case where the observation direction detection unit cannot detect the observation direction and the subject recognition unit detects a pre-registered subject from the latest captured image, the mode switching unit sets factors other than the observation direction to track the direction of the pre-registered subject and switches the immediate preceding mode to a third camera mode which is one of the other camera modes.

32. The wearable imaging device according to claim 31, wherein: In the case where the observation direction detection unit cannot detect the observation direction and the subject recognition unit cannot detect the same subject and the pre-registered subject, the mode switching unit sets the factors other than the observation direction to one of the observation direction detected before the observation direction detection unit lost the observation direction and the observation direction that was moving by a change amount before the observation direction detection unit lost the observation direction, and switches the immediately preceding mode to a fourth camera mode which is one of the other camera modes.

33. The wearable camera device according to claim 32, further comprising a field angle changing unit configured to change a field angle of the image output from the image output unit. in, In the fourth imaging mode, the field angle changing unit widens the field angle of the image in the recording direction compared to a prescribed field angle.

34. The wearable camera device according to claim 33, wherein: Even after the mode is switched to one of the first camera mode, the second camera mode, the third camera mode, and the fourth camera mode, the mode switching unit continues to function.

35. The wearable camera device according to claim 34, wherein: When the mode switching unit switches the fourth imaging mode to one of the first imaging mode, the second imaging mode, and the third imaging mode, the field angle changing unit returns the widened field angle to the prescribed field angle.

36. The wearable imaging device according to claim 33, further comprising a notification unit configured to notify a user of a detection error of the viewing direction when the viewing direction detection unit cannot detect the viewing direction. in, When the mode switching unit switches the first imaging mode to one of the other imaging modes, the notification unit notifies a user of the detection error.

37. The wearable imaging device according to claim 1, wherein: The viewing direction detection unit includes: a first viewing direction calculation unit configured to detect a facial direction of the user and calculate a first viewing direction according to the detected facial direction; a viewing direction estimating unit configured to estimate a second viewing direction based on factors other than the detected face direction; a reliability calculation unit configured to calculate the reliability of the first observation direction; and a viewing direction determination unit configured to determine the viewing direction, Wherein, when the reliability is equal to or greater than a threshold, the viewing direction determining unit determines the viewing direction as the first viewing direction, and Wherein, when the reliability is less than the threshold and the second viewing direction is reliable, the viewing direction determining unit determines the viewing direction based on the first viewing direction, the second viewing direction, and the reliability.

38. The wearable imaging device according to claim 1, wherein: The detection optical axis of the observation direction detection unit and the imaging optical axis of the imaging unit point in directions different from each other.

39. The wearable camera device according to claim 38, wherein: A detection optical axis of the viewing direction detection unit is directed from the viewing direction detection unit toward a jaw of a user.

40. The wearable camera device according to claim 38, wherein The imaging optical axis of the imaging unit is directed from the imaging unit to the front direction of the user.

41. The wearable imaging device according to claim 38, wherein The wearable imaging device in which the viewing direction detection unit and the imaging unit are integrally formed is built into a camera body, and When the user is wearing the camera body and viewing the wearable imaging device from the front, the total horizontal length of the wearable imaging device is longer than the total vertical length of the wearable imaging device.

42. The wearable imaging device according to claim 41, wherein: The camera body is provided with a fixing member for contacting a user's body, Here, when the user wears the camera body, the fixing members are respectively arranged near the right end and the left end of the wearable imaging device. 43 . The wearable imaging device according to claim 42 , further comprising a contact angle adjustment mechanism for adjusting an angle of the fixing member relative to a user's body.

44. The wearable imaging device according to claim 41, wherein The wearable camera device is connected to a neck suspension member, and the neck suspension member is used to wear the wearable camera device on the neck of a user. Here, when the user wears the camera body, the neck hanging member is connected to the vicinity of the right end and the left end of the wearable imaging device.

45. The wearable imaging device according to claim 44, wherein: The neck suspension member is provided with a neck suspension angle adjustment mechanism, and the neck suspension angle adjustment mechanism is used to adjust the angle of the neck suspension member relative to the wearable camera device.

46. ​​The wearable imaging device according to claim 44, wherein: The neck suspension member is provided with a belt portion whose cross-sectional shape is not a perfect circle, When the user wears the camera body, the distance between the right and left parts of the belt portion, which are symmetrical with respect to the wearable imaging device, becomes shorter from the lower side toward the upper side.

47. The wearable imaging device according to claim 44, wherein: The wearable camera device is connected to a power supply unit via the neck suspension member. Wherein, when the user wears the camera body, the power supply unit is arranged behind the user's neck.

48. The wearable imaging device according to claim 47, wherein: The wearable camera device is connected to the power supply unit via a power supply member. Wherein, the power supply member is arranged inside the neck suspension member.

49. The wearable imaging device according to claim 16, wherein: The imaging unit includes an imaging lens and an image sensor for converting an optical image formed by the imaging lens into raw data, wherein the imaging unit outputs the raw data read from a predetermined area of ​​the image sensor as an image captured by the imaging unit. The wearable camera device further includes an imaging unit configured to extract data within an area narrower than the predetermined area and including a target field of view in the recording direction and a margin around the target field of view from the raw data, and to develop the extracted data.

50. The wearable imaging device according to claim 49, wherein: The margin is a pixel area used by the image stabilization process.

51. The wearable imaging device according to claim 49, wherein: The developing unit changes the shape of the target field of view and the shape of the margin according to the recording direction and the optical properties of the imaging lens.

52. The wearable imaging device according to claim 49, wherein: The image output unit records the data extracted and developed by the developing unit as an image in the recording direction, and does not record data not extracted from the predetermined area.

53. The wearable camera device according to claim 52, wherein: The image output unit transmits the image of the recording direction to an external viewing device.

54. The wearable imaging device according to claim 53, wherein: The external viewing device applies optical correction processing and image stabilization processing to the image in the recording direction, and The image output unit transmits information required for the optical correction process and the image stabilization process together with the image in the recording direction.

55. The wearable camera apparatus according to claim 1, further being wirelessly connected to a portable device, the portable device comprising: a video file receiving unit configured to receive a video file in which metadata and an image of each frame are encoded for each frame, the metadata including in-image position information indicating a position and size of an image corresponding to a viewing direction of the user relative to the images of each frame captured by the wearable camera device; a first extraction unit configured to extract the metadata from the video file; a second extraction unit configured to extract an image of a frame encoded together with metadata extracted from the video file; a frame image correction unit configured to correct the image of the frame extracted by the second extraction unit using the metadata extracted by the first extraction unit; as well as A video recording unit is configured to record the frame image corrected by the frame image correction unit as a video image.

56. The wearable camera device according to claim 1, further connected to a calibrator wirelessly, the calibrator comprising: a first display unit configured to display positioning indicators captured by the camera unit of the wearable camera device during calibration; a face detection unit configured to detect a face of a user wearing the wearable camera device on his / her body; and A second display unit is configured to display a button that can be pressed to send an instruction of the calibration to the wearable camera device when it is determined based on the detection result of the face detection unit that the user is looking at the positioning index.

57. The wearable camera device according to claim 56, wherein the calibrator further comprises: Angular velocity sensor; as well as A second calibration unit is configured to calibrate the center position of the target field of view by comparing information from the angular velocity sensor with facial direction information about the user detected by the facial direction detection unit of the wearable camera device and wirelessly transmitted.

58. The wearable imaging device according to claim 56, wherein: the face detection unit captures a facial image of the user so that the facial image will include an imaging unit integrally mounted on the wearable imaging device worn on the user's body excluding the head, together with a viewing direction detection unit for detecting a viewing direction of the user, The calibrator further includes a vertical distance calculation unit configured to calculate a vertical distance between an imaging lens of the wearable imaging device and an eye position of a user based on a size of the imaging unit in the facial image.

59. A control method for a wearable camera device, the control method comprising: detecting a viewing direction of the user based on an image of the user's chin captured by an imager of a viewing direction detection unit worn around the user's neck; capturing images using a camera unit worn on the user's body; as well as outputting an image corresponding to the observation direction based on the captured image, The detection of the observation direction includes: irradiating the infrared irradiation surface including the user's chin with infrared light, detecting the reflected light of the infrared light reflected by the infrared irradiation surface using the imager, Output the user's viewing direction in the horizontal direction as the angle in the first detection direction, outputting the user's viewing direction in the vertical direction as an angle in a second detection direction perpendicular to the first detection direction, and The angle in the second detection direction is calculated based on the intensity of the reflected light at the user's chin position.

60. A non-transitory computer-readable storage medium storing a control program, the control program causing a computer to execute a control method for a wearable camera device, the control method comprising: detecting a viewing direction of the user based on an image of the user's chin captured by an imager of a viewing direction detection unit worn around the user's neck; capturing images using a camera unit worn on the user's body; as well as outputting an image corresponding to the observation direction based on the captured image, The detection of the observation direction includes: irradiating the infrared irradiation surface including the user's chin with infrared light, detecting the reflected light of the infrared light reflected by the infrared irradiation surface using the imager, Output the user's viewing direction in the horizontal direction as the angle in the first detection direction, outputting the user's viewing direction in the vertical direction as an angle in a second detection direction perpendicular to the first detection direction, and The angle in the second detection direction is calculated based on the intensity of the reflected light at the user's chin position.

Images