Imaging apparatus, control method thereof, calibration system, and storage medium

Abstract

The present application relates to an image pickup apparatus, a control method thereof, a calibration system, and a storage medium. The image pickup apparatus can eliminate a change in an image pickup direction by a user during an image pickup operation, and can easily obtain an image recording an experience while focusing attention on the experience. An observation direction detection unit is worn on a body part other than a head of a user, and detects an observation direction of the user. An image pickup unit is worn on the body and picks up an image. A recording direction determination unit determines a recording direction using a detection result of the observation direction detection unit. A deviation detection unit detects a deviation of the image pickup apparatus with respect to the body. An image recording unit records a part of the image picked up by the image pickup unit in a recording region determined in accordance with the recording direction and the deviation.

Description

Technical Field

[0001] This invention relates to a camera device used as a sports camera, a calibration system, a control method for the camera device, and a storage medium for storing the control program of the camera device. Background Technology

[0002] When a user is taking a picture of a subject with a camera, the user needs to continuously point the camera at the subject. Therefore, the user may find it difficult to manage actions other than the act of taking the picture. Furthermore, the user may find it difficult to focus their attention on their surroundings.

[0003] As a solution to these problems, a method is envisioned in which a camera is fixed to the user's head using a head-mounted accessory to capture images in the direction of observation. This allows the user to perform video recording without having to be preoccupied with the recording operation. Furthermore, a method for capturing images over a wide area using a celestial camera during the experience is also considered. This allows the user to focus on their experience during video recording. After the experience, the user can extract the desired image portion from the captured celestial images and edit that portion to obtain an image of the experience.

[0004] However, the former method requires cumbersome movements involving a moving camera on the head, which may make it difficult for users to concentrate on the experience. Furthermore, the latter method places a heavy burden on users because it requires them to extract and edit the desired portion from the entire celestial image.

[0005] Japanese Patent Application Publication No. 2007-74033 (JP 2007-74033A) discloses a technology that uses a first camera to photograph a subject and a second camera to photograph a user. This technology calculates the user's movement direction and gaze direction based on images captured by the second camera, determines the shooting direction of the first camera, and captures an image of the subject estimated based on the user's viewpoint and state.

[0006] Japanese Patent Application Publication No. 2017-60078 (JP 2017-60078A) (a family of U.S. Patent Application No. 20170085841) discloses an image recording system comprising a sensor device attached to a user's head and a camera device separately attached to the user's body or bag. The sensor device consists of a gyroscope sensor or an accelerometer sensor and detects the user's viewing direction. The camera device captures an image in the viewing direction detected by the sensor device.

[0007] However, since the second camera in JP 2007-74033A captures images of the user from a position far from the user, it requires high optical performance to calculate the user's movement direction and line of sight based on the images captured by the second camera. Furthermore, the device size increases due to the need for high processing power to process the images captured by the second camera.

[0008] Furthermore, since the sensor device in JP 2017-60078A directly detects the user's viewing direction, the user needs to equip the sensor device with their head, which does not solve the inconvenience of attaching any device to the head. Additionally, when the sensor device consists of a gyroscope or accelerometer, a certain level of accuracy can be achieved in detecting the relative viewing direction. However, since the accuracy in detecting the absolute viewing direction (especially the horizontal rotation direction) cannot be obtained, there are problems in practical applications. Summary of the Invention

[0009] This invention provides a technique that can eliminate changes in camera orientation by the user during video recording and can easily acquire images of the experience while the user focuses their attention on the experience.

[0010] Therefore, one aspect of the present invention provides a camera device comprising: an observation direction detection unit adapted to be worn on a body part of a user other than the head and configured to detect the user's observation direction; at least one camera unit adapted to be worn on the user's body and configured to capture an image; a recording direction determination unit configured to determine a recording direction using the detection result of the observation direction; a deviation detection unit configured to detect a deviation of the camera device relative to the user's body; and an image recording unit configured to record a portion of the image captured by the camera unit in a recording area determined according to the recording direction determined by the recording direction determination unit and the deviation detected by the deviation detection unit.

[0011] According to the present invention, it is unnecessary for the user to change the camera direction during the camera operation, and images of the experience can be easily obtained while focusing attention on the experience.

[0012] Further features of the invention will become apparent from the following description of typical embodiments with reference to the accompanying drawings. Attached Figure Description

[0013] Figure 1A This is an external view showing the camera body including the imaging / detection unit, which is a camera device according to the first embodiment.

[0014] Figure 1B This is a diagram showing the user wearing the camera body.

[0015] Figure 1C It shows from Figure 1A A diagram showing the battery unit inside the camera body as seen from the rear.

[0016] Figure 1D This is an external view of the display device, which is a portable device according to the first embodiment and is separate from the camera body.

[0017] Figure 2A This is a front view showing the imaging / detection unit inside the camera body.

[0018] Figure 2B This is a diagram showing the shape of the strap portion of the connecting member in the camera body.

[0019] Figure 2C This is a rear view showing the camera / detection unit.

[0020] Figure 2D This is a top view showing the camera / detection unit.

[0021] Figure 2E This is a diagram showing the structure of the face orientation detection unit arranged inside the camera / detection unit and below the face orientation detection window in the camera body.

[0022] Figure 2F This is a diagram showing the user wearing the camera body as seen from the user's left side.

[0023] Figure 3A , Figure 3B and Figure 3C This is a diagram showing the details of the battery cell.

[0024] Figure 4 This is a functional block diagram showing the camera body according to the first embodiment.

[0025] Figure 5 This is a block diagram illustrating the hardware structure of the camera body according to the first embodiment.

[0026] Figure 6 This is a block diagram showing the hardware structure of a display device.

[0027] Figure 7A This is a schematic flowchart illustrating the video recording / capture process performed by the camera body and the display device according to the first embodiment.

[0028] Figure 7B This illustrates the first embodiment. Figure 7A The flowchart of the preparation process subroutine in step S100.

[0029] Figure 7CThis illustrates the first embodiment. Figure 7A The flowchart of the face orientation detection processing subroutine in step S200.

[0030] Figure 7D This illustrates the first embodiment. Figure 7A The flowchart of the subroutine for determining the recording direction / region in step S300.

[0031] Figure 7E This illustrates the first embodiment. Figure 7A The flowchart of the image processing subroutine in step S500.

[0032] Figure 7F It is used to describe video image modes Figure 7A The diagram shows the processing steps S200 to S500.

[0033] Figure 8A This is a diagram showing the image of the user as seen from the face-oriented detection window.

[0034] Figure 8B This diagram illustrates the situation where fluorescent lights in a room appear as a background in the user's image as seen from the face detection window.

[0035] Figure 8C The image shown below illustrates the process of using an infrared detection device with its infrared LED off. Figure 8B The image shown is obtained by imaging the user and the fluorescent light in the background onto the sensor of the infrared detection device through the face orientation detection window.

[0036] Figure 8D The image shown below illustrates the process of creating an image with the infrared LED illuminated. Figure 8B The image shown is obtained by imaging the user and the fluorescent light in the background onto the sensor of the infrared detection device through the face orientation detection window.

[0037] Figure 8E It shows that by from Figure 8D Subtract from the image Figure 8C The image is a graph of the difference image calculated from the image.

[0038] Figure 8F The following graph shows the result, which is achieved by adjusting... Figure 8E The density of the difference image is obtained by scaling the light intensity of the reflected component of infrared light projected onto the user's face and neck.

[0039] Figure 8G Through in Figure 8FThe image is obtained by overlaying reference numerals to indicate body parts of the user, a double circle indicating the location of the throat, and a black circle indicating the location of the chin.

[0040] Figure 8H This shows how, when the user's face is turned to the right, the interaction with... Figure 8E A graph of the difference image calculated using a similar method.

[0041] Figure 8I This is a graph showing the results obtained through the following operations: Adjustment Figure 8H The image density is scaled to fit the intensity of the reflected component of infrared light projected onto the user's face and neck, and superimposed with a double circle indicating the location of the throat and a black circle indicating the location of the chin.

[0042] Figure 8J This is a diagram showing an image of a user with their face facing upwards at 33°, as seen from the face orientation detection window.

[0043] Figure 8K This is a diagram showing the result obtained by adjusting the angle of the user's face upwards by using the following method: Figure 8E A similar method calculates the density of the difference image to a scale suitable for the light intensity of the reflected component of infrared light projected onto the user's face and neck, and superimposes a double circle indicating the location of the throat and a black circle indicating the location of the chin.

[0044] Figure 9 This is a timing diagram showing the lighting timing of the infrared LED and related signals.

[0045] Figures 10A to 10D It is a diagram illustrating the vertical movement of the user's face.

[0046] Figure 11A This is a diagram showing the target field of view set in an ultra-wide-angle image captured by the camera unit of the camera body when the user is facing forward.

[0047] Figure 11B It shows from Figure 11A The image is a plot of the target field of view extracted from an ultra-wide-angle image.

[0048] Figure 11C This is a diagram showing the target field of view set in an ultra-wide-angle image when the user is observing subject A.

[0049] Figure 11D This demonstrates the extraction of data from ultra-wide-angle images. Figure 11C The image obtained by correcting the distortion and blur of the image in the target field of view.

[0050] Figure 11EIt shows that the user is using a ratio Figure 11C The field-angle set value is a small field-angle set value used to observe the target field of view in an ultra-wide-angle image when subject A is viewed.

[0051] Figure 11F This demonstrates the extraction of data from ultra-wide-angle images. Figure 11E The image obtained by correcting the distortion and blur of the image in the target field of view.

[0052] Figure 12A This is a diagram illustrating an example of a target field of view set in an ultra-wide-angle image.

[0053] Figure 12B This shows the field angle setting value and Figure 12A An example of the target field of view set in an ultra-wide-angle image with the same target field of view angle setting but different viewing directions.

[0054] Figure 12C This shows the field angle setting value and Figure 12A Another example of the target field of view set in an ultra-wide-angle image with the same target field of view angle setting but different viewing directions.

[0055] Figure 12D It shows the direction of observation and Figure 12C An example of the target field of view set in an ultra-wide-angle image where the target field of view is in the same viewing direction and the field angle setting value is smaller.

[0056] Figure 12E It is shown in Figure 12A The figure shows an example of an image stabilization margin given around the target field of view, corresponding to a predetermined image stabilization level.

[0057] Figure 12F It is shown in Figure 12B The target field of view shown is given with Figure 12E The figure shows an example of an image stability margin corresponding to the same image stability level.

[0058] Figure 12G It is shown in Figure 12D The target field of view shown is given with Figure 12E The figure shows an example of an image stability margin corresponding to the same image stability level.

[0059] Figure 13 This is a diagram showing a menu screen displayed on the display unit of a display device before the camera body operates to take a picture, displaying various settings for setting the video image mode.

[0060] Figure 14 It is shown Figure 7A The flowchart of the main record processing subroutine in step S600.

[0061] Figure 15 This is a diagram illustrating the data structure of the image file generated through master record processing.

[0062] Figure 16 yes Figure 7A The flowchart of the subroutine for sending processing to the display device in step S700.

[0063] Figure 17 It is shown Figure 7A The flowchart of the optical correction processing subroutine in step S800.

[0064] Figures 18A to 18F It is used for explanation Figure 17 The diagram shows the application of distortion correction in step S803.

[0065] Figure 19 It is shown Figure 7A The flowchart of the image stabilization subroutine in step S900.

[0066] Figure 20A and Figure 20B This is a diagram showing details of the calibrator used in the calibration process according to the second embodiment.

[0067] Figure 21 This is a flowchart illustrating the calibration process performed by the camera body and the calibrator according to the second embodiment.

[0068] Figure 22A This is shown during the calibration operation in the user's frontal orientation. Figure 21 The image shown on the display unit of the calibrator in step S3103.

[0069] Figure 22B It shows the user based on as Figure 22A The diagram shows the instructions indicating that the calibrator should be kept in the positive position.

[0070] Figure 22C It is shown in Figure 22B A schematic diagram of the entire ultra-wide-angle image captured by the camera lens in this state.

[0071] Figure 22D It shows that by... Figure 22C This is a schematic diagram of the image obtained by correcting aberrations in an ultra-wide-angle image.

[0072] Figure 22EThis is shown during the calibration operation in the user's frontal orientation. Figure 21 A schematic diagram of the facial orientation image obtained by the facial orientation detection unit in step S3108.

[0073] Figure 22F It is shown in Figure 21 A schematic diagram of the built-in camera image shown in step S3107.

[0074] Figure 23A This is shown during the calibration operation in the upper right direction for the user. Figure 21 The image shown on the display unit of the calibrator in step S3103.

[0075] Figure 23B It shows the user based on as Figure 23A The diagram shows the indicator indicating that the calibrator is held in the upper right position.

[0076] Figure 23C It is shown in Figure 23B A schematic diagram of the entire ultra-wide-angle image captured by the camera lens in this state.

[0077] Figure 23D It shows that by... Figure 23C This is a schematic diagram of the image obtained by correcting aberrations in an ultra-wide-angle image.

[0078] Figure 23E This indicates that during a calibration operation directed towards the user's upper right direction... Figure 21 A schematic diagram of the facial orientation image obtained by the facial orientation detection unit in step S3108.

[0079] Figure 24A and Figure 24B This is a schematic front view showing a user wearing the camera body.

[0080] Figure 25A and Figure 25B This is a schematic side view showing a user wearing the camera body.

[0081] Figure 26 This is a functional block diagram showing the camera body according to the third embodiment.

[0082] Figures 27A to 27C This is a diagram showing the image of the user as seen from the face-oriented detection window.

[0083] Figures 28A to 28C Each diagram shows the effective projection area of ​​the ultra-wide-angle image captured by the camera unit and the target field of view to be extracted by the image extraction / display unit.

[0084] Figure 29This is a flowchart illustrating the deviation detection process.

[0085] Figure 30A This is a diagram showing the user without tilting their head. Figure 30B This is a diagram showing the state of a user tilting their head horizontally (causing the head to tilt).

[0086] Figure 31 This is a functional block diagram showing the camera body according to the fourth embodiment.

[0087] Figure 32A and Figure 32B This is the user's image as seen from the facial detection window.

[0088] Figure 33A and Figure 33B Each diagram shows the effective projection area in the ultra-wide-angle image captured by the camera unit and the target field of view to be extracted by the image extraction / display unit.

[0089] Figure 34 This is a flowchart illustrating the process of tilt correction from facial tilt angle detection to the extracted image.

[0090] Figures 35A to 35C This is a schematic side view of a user wearing a camera body according to the fifth embodiment.

[0091] Figure 36A and Figure 36B This is a graph showing an example of the detection results for facial movement (tilt angle) in the vertical direction.

[0092] Figure 37 This is a diagram showing details of the calibrator used in the calibration process according to the sixth embodiment.

[0093] Figure 38 This is a flowchart illustrating the processing of the camera body in the calibration process according to the sixth embodiment.

[0094] Figure 39 This is a flowchart illustrating the calibrator's processing in the calibration process according to the sixth embodiment.

[0095] Figure 40A and Figure 40B This is a schematic diagram illustrating an example of an ultra-wide-angle image during calibration.

[0096] Figure 41A and Figure 41B This is a schematic diagram showing the display state of the display unit 803. Detailed Implementation

[0097] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0098] First, the first embodiment will be described. Figures 1A to 1D This diagram illustrates a camera system consisting of a camera body 1 and a display device 800 separate from the camera body 1. The camera body 1 includes a camera / detection unit 10, which is a wearable camera device according to a first embodiment. In this embodiment, although the camera body 1 and the display device 800 are separate devices, they can be integrated.

[0099] Figure 1A This is a diagram showing the exterior of the camera body 1. (Example) Figure 1A As shown, the camera body 1 includes an image / detection unit 10, a battery unit (power supply unit) 90, a right-side connecting member 80R, and a left-side connecting member 80L. The right-side connecting member 80R is located on the right side of the user's body. Figure 1A The left side of the device connects the camera / detection unit 10 and the battery unit 90. The left-side connecting member 80L is located on the left side of the user's body. Figure 1A (On the right side) connects the camera / detection unit 10 and the battery unit 90.

[0100] The camera / detection unit 10 is equipped with a face direction detection window 13, a start switch 14, a stop switch 15, a camera lens 16, an LED 17, and microphones 19L and 19R.

[0101] The face orientation detection window 13 allows the infrared LED 22 built into the camera / detection unit 10 to be used. Figure 5 Infrared light projected by the infrared irradiation unit is transmitted to detect the position of the user's facial features. The facial orientation detection window 13 also allows reflected infrared light from the face to pass through.

[0102] The start switch 14 is used to start the camera operation. The stop switch 15 is used to stop the camera operation. The camera lens 16 guides light from the subject to be photographed to the solid-state image sensor 42 inside the camera / detection unit 10. Figure 5 LED 17 indicates that camera operation is in progress. Additionally or optionally, LED 17 can be used as a warning light.

[0103] Microphones 19R and 19L pick up ambient sound. Microphone 19L picks up sound from the left side of the user's surroundings. Figure 1A The microphone 19R picks up sound from the right side of the user's surroundings. Figure 1A The sound on the left side of the middle.

[0104] Figure 1BThis diagram illustrates a user wearing the camera body 1. The user wears the camera body 1 with the battery unit 90 positioned near the user's back and the camera / detection unit 10 positioned near the user's front. Thus, the camera / detection unit 10 is located in front of the user's collarbone. At this time, the facial orientation detection window 13 is located below the user's jaw. The following description is arranged inside the facial orientation detection window 13. Figure 2E The infrared condenser lens 26 is shown. The optical axis (detection optical axis) of the infrared condenser lens 26 points towards the user's face, and in a direction different from the optical axis (imaging optical axis) of the camera lens 16. The face direction detection unit 20, including the infrared condenser lens 26, detects the user's viewing direction based on the position of the facial features. This allows the camera unit 40, described later, to capture an image of the subject in the viewing direction. Adjustments to the setting position due to individual differences in body shape and clothing will be explained later.

[0105] Furthermore, since the camera / detection unit 10 is arranged on the front of the body and the battery unit 90 is arranged on the back in this way, the weight of the camera body 1 is distributed, which reduces user fatigue and reduces the displacement of the camera body 1 caused by the force on the camera body 1 due to the user's movement.

[0106] Although an example has been described in which the user wears the camera body 1 in front of the user's collarbone with the camera / detection unit 10, this embodiment is not limited to this example. That is, the user can wear the camera body 1 anywhere on the user's body except the head, as long as the camera body 1 can detect the user's viewing direction and the camera unit 40 can capture an image of the subject in the viewing direction.

[0107] Figure 1C From Figure 1A The diagram shows the battery cell 90 as seen from the rear. (See diagram for reference.) Figure 1C As shown, the battery unit 90 is provided with a charging cable insertion slot 91, adjustment buttons 92L and 92R, and a spine avoidance cut 93.

[0108] A charging cable (not shown) can be connected to the charging cable insertion slot 91. External power is supplied to the internal batteries 4L and 94R (see [reference]) via the charging cable. Figure 3A It charges and supplies power to the camera / detection unit 10.

[0109] Adjustment buttons 92L and 92R are used to adjust the lengths of the strap portion 82L of the left connecting member 80L and the strap portion 82R of the right connecting member 80R. Adjustment button 92L is used to adjust the left strap portion 82L, and adjustment button 92R is used to adjust the right strap portion 82R. In this embodiment, although the lengths of the strap portions 82L and 82R are adjusted independently using adjustment buttons 92L and 92R, the lengths of the strap portions 82L and 82R can be adjusted simultaneously using a single button.

[0110] The spine avoidance cut 93 is formed by shaping the battery unit 90 so that the battery unit 90 will not come into contact with the spine. Since the spine avoidance cut 93 avoids the convex part of the body's spine, it reduces wearing discomfort and prevents the battery unit 90 from shifting laterally.

[0111] Figure 1D This is an external view showing the display device 800, a portable device according to the first embodiment, separate from the camera body 1. Figure 1D As shown, the display device 800 includes an A button 802, a display unit 803, a B button 804, a built-in camera 805, a face sensor 806, an angular velocity sensor 807, and an accelerometer 808. Furthermore, the display device 800 includes a wireless LAN unit enabling high-speed connection to the camera body 1. Figure 1D (Not shown in the image).

[0112] Button A 802 functions as the power button of display device 800. Display device 800 receives an ON / OFF operation by pressing and holding button A 802, and receives a specified processing timing by pressing button A 802 briefly.

[0113] The display unit 803 is used to examine the images captured by the camera body 1 and can display the menu screen required for setting. In this embodiment, a transparent touch sensor disposed on the surface of the display unit 803 receives touch operations on the displayed screen (e.g., the menu screen).

[0114] Button B 804 serves as calibration button 854 for the calibration process described later. The built-in camera 805 can capture images of a person observing the display device 800.

[0115] The face sensor 806 detects the facial shape and viewing direction of the person observing the display device 800. The specific structure of the face sensor 806 is not limited. For example, it can employ a structured optical sensor, a Time-of-Flight (ToF) sensor, or millimeter-wave radar.

[0116] Since the angular velocity sensor 807 is built into the display device 800, its meaning as a stereoscopic view is represented by dotted lines. Because the display device 800 in this embodiment also has the function of a calibrator described later, a three-axis gyroscope sensor is installed to detect the movement of the display device 800 as gyroscope data, enabling detection in the X, Y, and Z directions. An accelerometer 808 detects the posture of the display device 800.

[0117] It should be noted that a general smartphone is used as the display device 800 according to this embodiment. The camera system of this embodiment is implemented by matching the firmware in the smartphone with the firmware of the camera body 1. On the other hand, the camera system of this embodiment can be implemented by matching the firmware of the camera body 1 with the application and OS of the smartphone serving as the display device 800.

[0118] Figures 2A to 2F This is a diagram detailing the camera / detection unit 10. From... Figure 2A In the figures, components that have the same function as the parts already described are indicated by the same reference numerals, and the description of such components in this specification is omitted.

[0119] Figure 2A This is a front view showing the camera / detection unit 10. The right-side connecting member 80R has a strap portion 82R and an angle-holding member 81R made of rigid material for maintaining the angle relative to the camera / detection unit 10. The left-side connecting member 80L similarly has a strap portion 82L and an angle-holding member 81L.

[0120] Figure 2B This is a diagram showing the shapes of the belt portion 82L of the left connecting member 80L and the belt portion 82R of the right connecting member 80R. Figure 2B In the diagram, the angle-holding members 81L and 81R are shown as transparent members to show the shape of the strips 82L and 82R.

[0121] The strap 82L is positioned on the left side of the user's body when the user wears the camera body 1. Figure 2B The left side connection surface 83L and cable 84 are located on the right side of the camera body 1. The strap 82R is positioned on the right side of the user's body when the user wears the camera body 1. Figure 2B The right side of the left side of the middle is connected to the surface 83R.

[0122] The left connecting surface 83L connects to the angle-maintaining member 81L, and its cross-sectional shape is elliptical rather than circular. The right connecting surface 83R also has a similar elliptical shape. The right connecting surface 83R and the left connecting surface 83L are arranged symmetrically on both sides in an inverted V-shape. That is to say, in Figure 2BIn the middle, the distance between the right connecting surface 83R and the left connecting surface 83L decreases from the bottom to the top. As a result, since the long axis direction of the left connecting surface 83L and the right connecting surface 83R matches the user's body when the user is hanging the camera body 1, the straps 82L and 82R comfortably contact the user's body and prevent the camera / detection unit 10 from moving in the left-right and front-back directions.

[0123] Cable (power supply component) 84 is routed inside the strip 82L and electrically connects the battery unit 90 and the camera / detection unit 10. Cable 84 connects the power supply of the battery unit 90 to the camera / detection unit 10, or transmits electrical signals to external devices.

[0124] Figure 2C This is a rear view showing the camera / detection unit 10. Figure 2C Show the side that is in contact with the user's body. That is to say, Figure 2C From Figure 2A The diagram shows the view from the opposite side. Therefore, the positional relationship between the right connecting member 80R and the left connecting member 80L is... Figure 2A on the contrary.

[0125] The camera / detection unit 10 has a power switch 11, a camera mode switch 12, and chest contact pads 18a and 18b on its back side. The power switch 11 is used to switch the power supply of the camera body 1 on / off. Although the power switch 11 in this embodiment is a slider type, it is not limited to this. For example, the power switch 11 can be a push-button type switch, or it can be a switch integrated with the sliding cover (not shown) of the camera lens 16.

[0126] The camera mode switch (changing component) 12 is used to change the camera mode, that is, to change the mode related to camera operation. In this embodiment, the camera mode switch 12 can select the camera mode from still image mode, video image mode, and the preset mode set by the display device 800 as described below.

[0127] When the camera / detection unit 10 is attached to the user's body, the chest contact pads (fixed members) 18a and 18b touch the user's body. Figure 2AAs shown, the camera / detection unit 10 is configured such that its total lateral (left-right) length is longer than its total vertical (up-down) length when the camera body 1 is worn. Chest contact pads 18a and 18b are arranged near the right and left ends of the camera / detection unit 10, respectively. This arrangement reduces lateral rotational blur during camera operation. Furthermore, the chest contact pads 18a and 18b prevent the power switch 11 and camera mode switch 12 from touching the user's body. Additionally, the chest contact pads 18a and 18b prevent heat transfer to the user's body even if the camera / detection unit 10 becomes hot due to prolonged camera operation, and are used to adjust the angle of the camera / detection unit 10.

[0128] Figure 2D This is a top view showing the camera / detection unit 10. (See attached image.) Figure 2D As shown, a face direction detection window 13 is provided in the center of the top surface of the camera / detection unit 10, and chest contact pads 18a and 18b protrude from the camera / detection unit 10.

[0129] Figure 2E This diagram shows the structure of the face orientation detection unit 20, which is arranged inside the camera / detection unit 10 and below the face orientation detection window 13. The face orientation detection unit 20 is equipped with an infrared LED 22 and an infrared focusing lens 26. The face orientation detection unit 20 also includes the features described later. Figure 5 The infrared LED lighting circuit 21 and infrared detection device 27 are shown.

[0130] Infrared LED 22 projects infrared light towards the user 23 ( Figure 5 The infrared focusing lens 26 directs the reflected light 25 from the user when infrared light 23 is projected from the infrared LED 22. Figure 5 The image is projected onto the sensor (not shown) of the infrared detection device 27.

[0131] Figure 2F This diagram shows the user wearing the camera body 1 as seen from the user's left side. An angle adjustment button 85L is located in the angle holding member 81L and is used to adjust the angle of the camera / detection unit 10. The angle adjustment button (in...) Figure 2F (Not shown) is positioned symmetrically within the angle adjustment button 85L, which is located within the relative angle holding member 81R. Although in Figure 2A , Figure 2C and Figure 2D The angle adjustment buttons are actually visible, but they have been omitted for the sake of simplicity.

[0132] While pressing the angle adjustment button 85L Figure 2FWhen the angle-holding member 81L is moved upwards or downwards, the user can change the angle between the camera / detection unit 10 and the angle-holding member 81L. The right side is the same as the left side. In addition, the protrusion angles of the chest contact pads 18a and 18b can be changed. The functions of these two angle-changing members (angle adjustment button and chest contact pad) can adjust the camera / detection unit 10 so that the optical axis of the camera lens 16 is kept horizontally regardless of individual differences in chest position and shape.

[0133] Figure 3A , Figure 3B and Figure 3C This is a diagram showing details of battery cell 90. Figure 3A This is a partial perspective rear view showing the battery cell 90.

[0134] like Figure 3A As shown, the left battery 94L and the right battery 94R are symmetrically mounted within the battery unit 90 to maintain weight balance. Thus, because the left battery 94L and the right battery 94R are arranged symmetrically with respect to the center of the battery unit 90, weight balance in the left-right direction is achieved, and displacement of the camera body 1 is prevented. It should be noted that the battery unit 90 can hold a single battery.

[0135] Figure 3B This is a top view showing battery cell 90. Also in Figure 3B In the diagram, batteries 94L and 94R are shown as transparent components. (See diagram below.) Figure 3B In this way, since batteries 94L and 94R are symmetrically arranged on both sides of the spine avoiding the cut 93, the weight of the battery unit 90 can be reduced to alleviate the burden on the user.

[0136] Figure 3C This is a rear view of battery cell 90. Figure 3C This is a view from the side of the user's body that is being touched, that is, from... Figure 3A The image seen from the opposite side. For example... Figure 3C As shown, the spine is positioned centrally along the user's spine, avoiding the incision 93.

[0137] Figure 4 This is a functional block diagram showing the camera body 1. In the following text, it will be used... Figure 4 This is a brief explanation of the processing performed by the camera body 1. Details will follow.

[0138] like Figure 4 As shown, the camera body 1 is equipped with a face orientation detection unit 20, a recording direction / field angle determination unit 30, an imaging unit 40, an image extraction / display unit 50, a main recording unit 60, a transmission unit 70, and a second controller 111. These functional blocks are controlled by a general control CPU 101 that controls the entire camera body 1. Figure 5This is achieved through control.

[0139] The facial orientation detection unit 20 (observation orientation detection unit) is a functional block executed by the aforementioned infrared LED 22, infrared detection device 27, etc. The facial orientation detection unit 20 estimates the observation orientation by detecting the facial orientation and transmits the observation orientation to the recording orientation / field angle determination unit 30.

[0140] The recording direction / field angle determination unit (recording direction determination unit) 30 determines information related to the position and region of the image to be extracted from the camera data output from the camera unit 40 by performing various calculations based on the observation direction estimated by the face direction detection unit 20. This information is then passed to the image extraction / display unit 50.

[0141] The camera unit 40 comprises a camera driver 41, a solid-state image sensor 42, and an image signal processing circuit 43, which will be described later. Figure 5 The imaging unit 40 consists of components such as a light beam from the subject, which forms an image on the solid-state image sensor 42, and the signal obtained through photoelectric conversion is output to the image signal processing circuit 43. Then, the image signal processing circuit 43 generates captured image data based on the signal from the solid-state image sensor 42, and transmits the captured image data to the image extraction / display unit 50.

[0142] The image extraction / display unit (display unit) 50 extracts the image data viewed by the user from the captured image data transmitted from the imaging unit 40 using information transmitted from the recording direction / field angle determination unit 30. Then, the image extraction / display unit 50 displays the extracted image and transmits the displayed image to the main recording unit 60.

[0143] The main record unit 60 is composed of the main memory 103 ( Figure 5 Functional blocks, such as ) are used to record image data and transmit the image data to the sending unit 70 at the required time intervals.

[0144] Transmitting unit 70 and display device 800 ( Figure 1D It wirelessly connects to predetermined communication partners such as the calibrator 850 and the simplified display device 900, and communicates with these communication partners.

[0145] The display device 800 can be connected to the transmitting unit 70 via a high-speed wireless LAN (hereinafter referred to as a "high-speed wireless network"). In this embodiment, the high-speed wireless network adopts wireless communication corresponding to the IEEE 802.11ax (WiFi 6) standard. On the other hand, wireless communication corresponding to other standards such as WiFi 4 and WiFi 5 can be adopted. Furthermore, the display device 800 can be a dedicated device developed for the camera body 1, or it can be a general smartphone, tablet terminal, etc.

[0146] Additionally, the display device 800 can connect to the transmitting unit 70 via a low-power wireless network, via both a high-speed wireless network and a low-power wireless network, or by switching networks simultaneously. In this embodiment, a large amount of data is transmitted via a high-speed wireless network, while a small amount of data and data that does not require rapid transmission are transmitted via a low-power wireless network. Although Bluetooth (registered trademark) is used for the low-power wireless network in this embodiment, other short-range wireless communications such as NFC (Near Field Communication) can be employed.

[0147] The calibrator 850 performs initial and individual setup of the camera body 1, and can be connected to the transmitting unit 70 via a high-speed wireless network in the same manner as the display device 800. Details of the calibrator 850 will be described later. Furthermore, the display device 800 may also have the functionality of the calibrator 850.

[0148] The simplified display device 900 can connect to the transmitting unit 70, for example, via a low-power wireless network. Although the simplified display device 900 cannot communicate large amounts of data with the transmitting unit 70 due to time constraints, it can transmit camera start / stop timing and can be used for image inspection at the composition check level. Furthermore, the simplified display device 900 can be a dedicated device developed for the camera body 1, like the display device 800, or it can be a smartwatch, etc.

[0149] Figure 5 This is a block diagram showing the hardware structure of the camera body 1. Furthermore, using... Figures 1A to 1C The structures and functions described are indicated by the same reference numerals, and detailed descriptions of these structures and functions will be omitted.

[0150] like Figure 5 As shown, the camera body 1 is equipped with a general control CPU 11, a power switch 11, a camera mode switch 12, a face orientation detection window 13, a start switch 14, a stop switch 15, a camera lens 16, and an LED 17.

[0151] The camera body 1 is also provided with a face orientation detection unit 20. Figure 4 The infrared LED lighting circuit 21, infrared LED 22, infrared focusing lens 26, and infrared detection device 27 are included.

[0152] In addition, the camera body 1 is provided with an image capturing unit 40. Figure 5 ) and transmitting unit 70 ( Figure 5 (It consists of a low-power wireless communication unit 71 and a high-speed wireless communication unit 72).

[0153] The camera unit 40 includes a camera driver 41, a solid-state image sensor 42, and an image signal processing circuit 43. The camera driver 41 includes a timing generator, which generates various timing signals, outputs these signals to various parts related to camera operation, and drives the solid-state image sensor 42. The solid-state image sensor 42 will transmit the signal through the camera's signal processing circuitry. Figure 1A The image of the subject formed by the camera lens 16 is converted into a signal by photoelectric conversion and output to the image signal processing circuit 43. The image signal processing circuit 43 outputs the image data generated by applying clamping processing and A / D conversion processing to the signal from the solid-state image sensor 42 to the overall control CPU 101.

[0154] In this embodiment, although the camera body 1 has a single camera unit 40, the camera body 1 may have two or more camera units to capture 3D images, capture images with a wider field of view than those obtained by a single camera unit, or capture images in different directions.

[0155] The camera body 1 is equipped with various memory, such as a large-capacity non-volatile memory 51, an internal non-volatile memory 102, and a main memory 103.

[0156] In addition, the camera body 1 is equipped with an audio processor 104, a speaker 105, a vibrator 106, an angular velocity sensor 107, an acceleration sensor 108, and various switches 110.

[0157] The above uses Figure 2C The switch, such as power switch 11, is connected to the overall control CPU 101. The overall control CPU 101 controls the entire camera body 1. Figure 4 The recording direction / field angle determination unit 30, the image extraction / display unit 50, and the second controller 111 are implemented by the overall control CPU 101.

[0158] Infrared LED lighting circuit 21 controls the above applications Figure 2E The infrared LED 22 is lit to control the projection of infrared light 23 from the infrared LED 22 toward the user.

[0159] The face orientation detection window 13 is composed of a visible light cutoff filter that almost prevents visible light from passing through, while fully allowing infrared light 23 belonging to the infrared region and its reflected light 25 to pass through.

[0160] The infrared focusing lens 26 focuses the reflected light 25.

[0161] The infrared detection device (infrared detection unit) 27 has a sensor that detects the reflected light 25 focused by the infrared focusing lens 26. The sensor converts the image formed by the focused reflected light 25 into sensor data and transmits the sensor data to the overall control CPU 101.

[0162] like Figure 1B As shown, when the user is wearing the camera body 1, the face orientation detection window 13 is located below the user's jaw. Therefore, as Figure 5 As shown, infrared light 23 projected from infrared LED 22 passes through facial direction detection window 13 and illuminates infrared illumination surface 24 near the user's jaw. Furthermore, reflected light 25 reflected from infrared illumination surface 24 passes through facial direction detection window 13 and is focused by infrared focusing lens 26 onto the sensor in infrared detection device 27.

[0163] exist Figures 1A to 1C Various switches 110 are not shown. Although details are omitted, the various switches 110 are used to perform functions unrelated to this embodiment.

[0164] The internal non-volatile memory 102 is composed of flash memory or the like and stores the boot program of the overall control CPU 101, as well as the setting values ​​of various program modes. In this embodiment, the setting values ​​of the observation field of view (field angle) and the setting values ​​of the image stabilization effect level are recorded.

[0165] The main memory 103 is composed of RAM and the like, and temporarily stores the image data being processed and the calculation results of the overall control computer 101.

[0166] The high-capacity non-volatile memory 51 stores image data. In this embodiment, the high-capacity non-volatile memory 51 is a non-removable semiconductor memory. However, the high-capacity non-volatile memory can be constructed from a removable storage medium such as an SD card and can be used in conjunction with the internal non-volatile memory 102.

[0167] The low-power wireless communication unit 71 exchanges data with the display device 800, the calibrator 850, and the simplified display device 900 via a low-power wireless network. The high-speed wireless communication unit 72 exchanges data with the display device 800 and the calibrator 850 via a high-speed wireless network.

[0168] The audio processor 104 processes external sound (analog signal) collected by the left microphone 19L and the right microphone 19R and generates an audio signal.

[0169] LED 17, speaker 105, and vibrator 106 are warning components that alert the user by emitting light, sound, and vibration. These warning components are used to notify the user of the status of the camera body 1 or to alert the user that a warning has been received.

[0170] The angular velocity sensor 107 uses a gyroscope or similar device and detects the movement of the camera body 1 itself as gyroscope data. The accelerometer sensor 108 detects the posture of the camera / detection unit 10.

[0171] Figure 6 This is a block diagram illustrating the hardware structure of the display device 800. (Using...) Figure 1D The components are indicated by the same reference numerals, and descriptions of these components will be omitted for the sake of simplicity.

[0172] like Figure 6 As shown, the display device 800 is equipped with a display device controller 801, an A button 802, a display unit 803, a B button 804, a built-in camera 805, a face sensor 806, an angular velocity sensor 807, an acceleration sensor 808, an image signal processing circuit 809, and various switches 811.

[0173] In addition, the display device 800 includes an internal non-volatile memory 812, a main memory 813, a large-capacity non-volatile memory 814, a speaker 815, a vibrator 816, an LED 817, an audio processor 820, a low-power wireless communication unit 871, and a high-speed wireless communication unit 872. These components are connected to a display device controller 801. The display device controller 801 is composed of a CPU and controls the display device 800.

[0174] The image signal processing circuit 809 performs the same functions as the image driver 41, solid-state image sensor 42, and image signal processing circuit 43 inside the camera body 1. The image signal processing circuit 809, together with the built-in camera lens 805a, constitutes... Figure 1D The built-in camera 805. The display device controller 801 processes the data output from the image signal processing circuit 809. The data processing will be described later.

[0175] Various switches 811 are used to perform functions unrelated to this embodiment. The angular velocity sensor 807 uses a gyroscope or similar device to detect movement of the display device 800.

[0176] Accelerometer 808 detects the posture of display device 800 itself. Angular velocity sensor 807 and accelerometer 808 are built into display device 800 and have the same functions as angular velocity sensor 107 and accelerometer 108 in camera body 1 described above.

[0177] The internal non-volatile memory 812 is composed of flash memory and the like, and stores the boot program of the display device controller 801, as well as the setting values ​​of various program modes.

[0178] The main memory 813 is composed of RAM and other components, and temporarily stores the image data being processed and the calculation results of the image signal processing circuit 809.

[0179] The large-capacity non-volatile memory 814 stores image data of the display device 800. In this embodiment, the large-capacity non-volatile memory 814 is composed of a removable memory such as an SD card. It should be noted that, similar to the large-capacity non-volatile memory 51 in the camera body 1, the large-capacity non-volatile memory 814 can be composed of a fixed memory.

[0180] In order to notify the user of the status of the display device 800 and to warn the user, the speaker 815 outputs sound, the vibrator 816 vibrates, and the LED 817 illuminates.

[0181] The audio processor 820 processes external sound (analog signal) collected by the left microphone 819L and the right microphone 819R and generates an audio signal.

[0182] The low-power wireless communication unit 871 exchanges data with the camera body 1 via a low-power wireless network. The high-speed wireless communication unit 872 exchanges data with the camera body 1 via a high-speed wireless network.

[0183] The face sensor (face detection unit) 806 is equipped with an infrared LED lighting circuit 821, an infrared LED 822, an infrared focusing lens 826, and an infrared detection device 827.

[0184] The infrared LED lighting circuit 821 has the same characteristics as... Figure 5 The infrared LED lighting circuit 21 has the same function as the infrared LED 822, and controls the lighting of the infrared LED 822 to control the projection of the infrared light 823 from the infrared LED 822 toward the user. The infrared focusing lens 826 focuses the reflected light 825 of the infrared light 823.

[0185] The infrared detection device 827 has a sensor that detects the reflected light 825 focused by the infrared focusing lens 826. The sensor converts the focused reflected light 825 into sensor data and transmits the sensor data to the display device controller 801.

[0186] exist Figure 1D When the face sensor 806 shown is pointed at the user, such as Figure 6 As shown, infrared light 823 projected from infrared LED 822 illuminates the infrared illumination surface 824, which serves as the user's entire face. Furthermore, reflected light 825 reflected from the infrared illumination surface 824 is focused by infrared focusing lens 826 onto a sensor in detection device 827.

[0187] Other functions 830 are functions of a smart phone, such as telephone functions, which are unrelated to this embodiment.

[0188] The following will explain how to use the camera body 1 and the display device 800. Figure 7A This is a schematic flowchart illustrating the video recording / capture process performed by the camera body 1 and the display device 800 according to the first embodiment.

[0189] To aid in the description, Figure 7A The right side of each step shows the unit that performs the processing in each step. Figure 4 and Figure 5 The reference numerals shown in the accompanying drawings. That is to say, Figure 7A Steps S100 to S700 are executed by the camera body 1, and Figure 7A Steps S800 to S1000 are executed by the display device 800.

[0190] When the power switch 11 is set to ON and the power to the camera body 1 is turned on, the overall control CPU 101 is activated and reads the boot program from the internal non-volatile memory 102. Then, in step S100, the overall control CPU 101 performs preparation processing to set up the camera body 1 before the recording operation. This will be used later. Figure 7B To explain the details of the preparation process.

[0191] In step S200, facial orientation detection processing is performed to estimate the observation orientation based on the facial orientation detected by the facial orientation detection unit 20. This will be used later. Figure 7C To explain the details of the face orientation detection process, this process is performed at a predetermined frame rate.

[0192] In step S300, the recording direction / field angle determination unit 30 performs the recording direction / region determination process. This will be used later. Figure 7D To explain the details of the recording direction / area determination process.

[0193] In step S400, the camera unit 40 captures images and generates camera data.

[0194] In step S500, the image extraction / development unit 50 extracts an image from the camera data generated in step S400 based on the recording direction / field angle information determined in step S300, and performs recording area development processing to develop the extracted area. This will be used later. Figure 7E To put it simply, it's like dealing with the details.

[0195] In step S600, the main recording unit (image recording unit) 60 performs main recording processing, storing the data developed in step S500 as image data in the main memory 103. This will be used later. Figure 14 This will explain the details of the master record processing.

[0196] In step S700, the transmitting unit 70 performs a transmission process to the display device 800, which wirelessly transmits the image data recorded in step S600 at a specified time interval to the display device 800. This will be used later. Figure 16 To explain the details of the transmission process to display device 800.

[0197] The steps beginning at step S800 are performed by the display device 800. In step S800, the display device controller 801 performs an optical correction process that corrects optical aberrations in the image transmitted from the camera body 1 in step S700. This will be used later. Figure 17 To explain the details of the optical correction process.

[0198] In step S900, the display device controller 801 applies image stabilization processing to the image whose optical aberrations were corrected in step S800. This will be used later. Figure 19 The details of the image stabilization process will now be explained. It should be noted that the order of steps S800 and S900 can be reversed. That is, image stabilization can be performed beforehand and optical correction can be performed afterward.

[0199] In step S1000, the display device controller (video recording unit) 801 performs auxiliary recording processing, which records the image, for which the optical correction processing in step S800 and the image stabilization processing in step S900 have been applied, into a large-capacity non-volatile memory 814. Then, the display device controller 801 terminates the processing.

[0200] Next, we will use Figures 7B to 7F Other accompanying figures are described in detail in the order of processing. Figure 7A Subroutines in each step. Figure 7B It is shown Figure 7A The flowchart of the preparation processing subroutine in step S100 is shown below. In the following text, it will be used... Figures 2A to 2F and Figure 5The components shown illustrate the process.

[0201] In step S101, it is determined whether the power switch 11 is turned on. If the power is off, the process waits. If the power is turned on, the process proceeds to step S102.

[0202] In step S102, the mode selected by the camera mode switch 12 is determined. As a result of this determination, if the mode selected by the camera mode switch 12 is video image mode, the process proceeds to step S103.

[0203] In step S103, various setting values ​​for the video image mode are read from the internal non-volatile memory 102 and stored in the main memory 103. Then, the process proceeds to step S104. The various setting values ​​for the video image mode include the field angle setting value V. ang And image stabilization level. In this embodiment, the field angle setting value V ang The angle is preset to 90°. The image stabilization level is selected from "Strong", "Medium", and "OFF". In step S104, operation of the camera driver 41 used for the video image mode begins. Then, the process exits from this subroutine.

[0204] As a result of the determination in step S102, if the mode selected by the camera mode switch 12 is still image mode, the process proceeds to step S106. In step S106, various setting values ​​for still image mode are read from the internal non-volatile memory 102 and stored in the main memory 103. Then, the process proceeds to step S107. The various setting values ​​for still image mode include the field angle setting value V. ang And image stabilization level. In this embodiment, the field angle setting value V ang The angle is preset to 45°. The image stabilization level is selected from "Strong", "Medium", and "Off". In step S107, operation of the camera driver 41 used for still image mode is started. Then, the process exits from this subroutine.

[0205] As a result of the determination in step S102, if the mode selected by the camera mode switch 12 is the preset mode, the process proceeds to step S108. The preset mode is a third mode other than the video image mode and the still image mode. In the preset mode, the camera mode of the camera body 1 can be set by an external device such as the display device 800. The preset mode is a mode used to customize camera operation.

[0206] In addition to the field angle, the preset mode may also include the image stabilization level (selected from "strong", "medium" and "off") and speech recognition settings not described in this embodiment.

[0207] In step S108, various setting values ​​for the preset mode are read from the internal non-volatile memory 102 and stored in the main memory 103. Then, the process proceeds to step S109. The various setting values ​​for the preset mode include the field angle setting value V. ang And the image stabilization level selected from "Strong", "Medium" and "Off".

[0208] In step S109, the operation of the camera driver 41 used in the preset mode begins. Then, the process exits from this subroutine.

[0209] After that, it will be used Figure 13 This will explain the various setting values ​​of the video image mode read in step S103. Figure 13 This is a diagram showing a menu screen displayed on the display unit 803 of the display device 800 before the camera body 1 performs its recording operation, displaying various setting values ​​for setting the video image mode. Figure 1D The components are denoted by the same reference numerals, and descriptions of these components will be omitted. The display unit 803 has a touch panel function, and will be described assuming that the display unit 803 functions via touch operations such as swiping.

[0210] like Figure 13 As shown, the menu screen includes a preview screen 831, a zoom lever 832, a recording start / stop button 833, a power switch 834, a battery level indicator 835, a button 836, a lever 837, and an icon display area 838. Users can check the image, zoom level, and field of view captured by the camera body 1 on the preview screen 831.

[0211] Users can change the zoom setting by moving the zoom lever 832 to the right or left. This embodiment illustrates that the field angle setting value V can be selected from 45°, 90°, 110°, and 130°. ang On the other hand, the field angle setting value V can be adjusted by operating the zoom lever 832. ang Set it to a value other than these four.

[0212] The recording start / stop button 833 is a toggle switch that functions as both the start switch 14 and the stop switch 15. Switch 834 toggles between "off" and "on" image stabilization. The battery level indicator 835 displays the battery level of the camera body 1. Button 836 is used to change modes.

[0213] Rod 837 is used to set the image stabilization level. In this embodiment, although the image stabilization level can be set to "strong" or "medium," another image stabilization level (e.g., "weak") can also be set. Furthermore, the image stabilization level can be set in a stepless manner. Multiple thumbnail icons used for previewing are displayed in icon display area 838.

[0214] Figure 7C It is shown Figure 7A The flowchart shows the subroutine for face orientation detection processing in step S200. Before explaining the details of this processing, we will use... Figures 8A to 8K This will illustrate a facial orientation detection method using infrared light.

[0215] Figure 8A This is a diagram showing a visible light image of a user's face viewed from the position of the face-direction detection window 13. Figure 8A The image is the same as the image captured by the visible light image sensor under the assumption that visible light is allowed to pass through the face direction detection window 13 and the visible light image sensor is installed as a sensor of the infrared detection device 27.

[0216] Figure 8A The image includes the front of the neck 201 above the user's collarbone, the root of the jaw 202, the chin 203, and the face 204 including the nose. Figure 8B It is shown in Figure 8A The image shown depicts a fluorescent lamp 205 in a room appearing as a background in a user's visible light image.

[0217] exist Figure 8B In the visible light image, fluorescent lights 205 appear around the user. Thus, due to the various backgrounds appearing in the user's image depending on usage conditions, it becomes difficult for the face orientation detection unit 20 or the overall control CPU 101 to extract the face image from the visible light image. On the other hand, although techniques exist for extracting such images using AI, these techniques are unsuitable for the camera body 1 of a portable device because the overall control CPU 101 requires high performance.

[0218] Therefore, the camera body 1 in the first embodiment uses infrared images to detect the user's face. Since the face orientation detection window 13 is composed of a visible light cutoff filter, most visible light is blocked. Therefore, the image obtained by the infrared detection device 27 differs from... Figure 8A and Figure 8B The image in the image.

[0219] Figure 8C This is a diagram showing an infrared image, which is obtained by displaying the infrared LED 22 in a state where it is not lit, and... Figure 8BThe user and the fluorescent light in the background are imaged onto the sensor of the infrared detection device 27 through the face orientation detection window 13.

[0220] exist Figure 8C In the infrared image, the user's neck and jaw are dark. On the other hand, since the fluorescent lamp 205 emits an infrared component in addition to visible light, it is slightly brighter.

[0221] Figure 8D The image shown below is obtained by displaying the image with the infrared LED 22 lit, and... Figure 8B The user and the fluorescent light in the background are imaged onto the sensor of the infrared detection device 27 through the face orientation detection window 13.

[0222] exist Figure 8D In the image, the user's neck and jaw are bright because infrared light is reflected. On the other hand, with Figure 8C In contrast, the brightness around fluorescent lamp 205 remained unchanged.

[0223] Figure 8E It shows that by from Figure 8D Subtract from the image Figure 8C The image is a graph of the difference image calculated from the image. The user's face appears.

[0224] In this way, the overall control CPU (image acquisition unit) 101 obtains a difference image (hereinafter referred to as a facial image) by calculating the difference between the image formed on the sensor of the infrared detection device 27 when the infrared LED 22 is lit and the image formed on the sensor when the infrared LED 22 is not lit.

[0225] In this embodiment, the facial orientation detection unit 20 uses a method to obtain a facial image by extracting the infrared reflection intensity as a two-dimensional image using an infrared detection device 27. The sensor of the infrared detection device 27 adopts a structure similar to that of a general image sensor and obtains facial images frame by frame. The vertical synchronization signal (hereinafter referred to as the V signal) for obtaining frame synchronization is generated by the infrared detection device 27 and output to the overall control CPU 101.

[0226] Figure 9 This is a timing diagram showing the timing of the infrared LED 22 turning on and off, as well as the related signals.

[0227] exist Figure 9The image data is shown in the following order from top to bottom: the V signal output from the infrared detection device 27, the H position of the image signal output from the sensor of the infrared detection device 27, the IR-ON (IR-on) signal output from the overall control CPU 101 to the infrared LED lighting circuit 21, and the camera data output from the sensor of the infrared detection device 27 to the overall control CPU 21. The horizontal axis of these four signals is the same time axis. When the V signal goes high, the timing for frame synchronization and the timing for lighting up and turning off the infrared LED 22 are obtained.

[0228] Figure 9 The time period t1 for obtaining the first facial image and the time period t2 for obtaining the second facial image are shown.

[0229] Infrared detection device 27 controls the operation of the sensor, so that, Figure 9 As shown, the H position of the image signal will be synchronized with the V signal. Since the sensor of the infrared detection device 27, as described above, adopts a structure similar to that of a general image sensor and its operation is well known, a detailed description of the control method is omitted.

[0230] The overall control CPU 101 synchronously controls the switching of the IR-ON signal between high and low, in accordance with the V signal. Specifically, the overall control CPU 101 outputs a low IR-ON signal to the infrared LED lighting circuit 21 during time period t1, and outputs a high IR-ON signal to the infrared LED lighting circuit 21 during the second time period t2.

[0231] During the high-frequency period of the IR-ON signal, the infrared LED lighting circuit 21 illuminates the infrared LED 22 to project infrared light 23 towards the user. Conversely, during the low-frequency period of the IR-ON signal, the infrared LED lighting circuit 21 turns off the infrared LED 22.

[0232] The vertical axis of the camera data represents the signal intensity as the amount of light received by the reflected light 25. Since the infrared LED 22 is off during the first time period t1, there is no reflected light from the user's face, and the received light is as follows: Figure 8C The image data shown is as follows. On the other hand, since the infrared LED 22 is lit during the second time period t2, reflected light 25 is generated from the user's face, and as shown in the image data... Figure 8D The image data shown indicates that the signal strength in time period t2 is increased by 25% due to reflected light from the user's face compared to the signal strength in time period t1.

[0233] By subtracting the image data captured during the first time period t1 from the image data captured during the second time period t2, we obtained... Figure 9 The lower part of the facial image is shown. As a result of subtraction, the following is obtained: Figure 8EThe image data shown is a facial image data in which only the component of reflected light 25 from the user's face is extracted.

[0234] Figure 7C The diagram shows the use of Figures 8C to 8E and Figure 9 The facial orientation detection process in step S200 of the operation.

[0235] In step S201, when the V signal output from the infrared detection device 27 goes high, a timing V1 starting from the first time period t1 is obtained. After obtaining the timing V1, the process proceeds to step S202.

[0236] In step S202, the IR-ON signal is set low and output to the infrared LED lighting circuit 21. As a result, the infrared LED 22 is turned off.

[0237] In step S203, a frame of image data output from the infrared detection device 27 during the first time period t1 is read. This image data is temporarily stored as Frame 1 in the main memory 103.

[0238] In step S204, when the V signal output from the infrared detection device 27 goes high, a timing V2 is obtained at the start of the second time period t2. Upon obtaining timing V2, the process proceeds to step S205.

[0239] In step S205, the IR-ON signal is set high and output to the infrared LED lighting circuit 21. As a result, the infrared LED 22 is lit.

[0240] In step S206, a frame of image data output from the infrared detection device 27 during the second time period t2 is read. This image data is temporarily stored as frame 2 in the main memory 103.

[0241] In step S207, the IR-ON signal is set low and output to the infrared LED lighting circuit 21. As a result, the infrared LED 22 is turned off.

[0242] In step S208, frames 1 and 2 are read from main memory 103, and the value from frame 1 is calculated by subtracting frame 1 from frame 2. Figure 9 The facial image shown corresponds to the light intensity Fn of the user's reflected light 25. This process is commonly referred to as black subtraction.

[0243] In step S209, the throat position (neck rotation center) is extracted from the light intensity Fn. First, the overall control CPU (segmentation unit) 101 segments the facial image based on the light intensity Fn into segments to be used... Figure 8F The description includes multiple distance regions.

[0244] Figure 8F It shows how to adjust Figure 8E The image shown is a result obtained by scaling the intensity of the difference image to fit the intensity of the reflected light 25 of the infrared light 23 projected onto the user's face and neck. Figure 8F The light intensity distribution is shown in relation to various parts of the user's face and neck.

[0245] Figure 8F The facial image on the left is shown by applying gray steps to each segmented region. Figure 8E The intensity distribution of reflected light 25 in the facial image shown. The Xf axis is oriented in a direction from the center of the user's neck toward the chin.

[0246] exist Figure 8F In the graph on the right, the horizontal axis represents the light intensity on the Xf axis of the facial image, and the vertical axis represents the Xf axis. The light intensity, as shown by the horizontal axis, increases as you move to the right.

[0247] Based on light intensity, Figure 8F The facial image is segmented into six regions (distance regions) 211 to 216. Region 211 is the region with the strongest light intensity and is shown as white in grayscale. Region 212 is the region with slightly lower light intensity compared to region 211 and is shown as a very bright gray in grayscale. Region 213 is the region with a further decrease in light intensity compared to region 212 and is shown as a light gray in grayscale. Region 214 is the region with a further decrease in light intensity compared to region 213 and is shown as a medium gray in grayscale. Region 215 is the region with a further decrease in light intensity compared to region 214 and is shown as a slightly darker gray in grayscale. Region 216 is the region with the weakest light intensity and is shown as the darkest gray in grayscale. The area above region 216 is shown as black, which does not show any light intensity.

[0248] Will use Figures 10A to 10D Let's explain the light intensity in detail. Figures 10A to 10D It is a diagram illustrating the vertical movement of the user's face and showing the state as viewed from the user's left side.

[0249] Figure 10AThis is a diagram showing the state where the user is facing forward. There is a camera / detection unit 10 in front of the user's collarbone. In addition, infrared light 23 from an infrared LED 22 irradiates the lower part of the user's head from a face direction detection window 13 installed at the upper part of the camera / detection unit 10. The distance Dn from the face direction detection window 13 to the throat 200 above the user's collarbone, the distance Db from the face direction detection window 13 to the root 202 of the jaw, and the distance Dc from the face direction detection window 13 to the chin 203 satisfy the relationship Dn < Db < Dc. Since the light intensity is inversely proportional to the square of the distance, the light intensity in the image formed on the sensor by the reflected light 25 from the infrared irradiation surface 24 gradually weakens in the order of the throat 200, the root 202 of the jaw, and the chin 203. In addition, since the distance from the face direction detection window 13 to the face 204 including the nose is still longer than the distance Dc, the light intensity in the image corresponding to the face 204 becomes weaker. That is, in Figure 10A the shown case, an image with Figure 8F the shown light intensity distribution is obtained.

[0250] It should be noted that the configuration of the face direction detection unit 20 is not limited to the configuration shown in this embodiment, as long as the face direction of the user can be detected. For example, an infrared pattern can be projected from an infrared LED (infrared pattern irradiation unit) 22, and a sensor (infrared pattern detection unit) of an infrared detection device 27 can detect the infrared pattern reflected from the irradiation target. In addition, the sensor of the infrared detection device 27 can be a sensor (infrared phase comparison unit) that compares the phase of the infrared light 23 and the phase of the reflected light 25. For example, a ToF (time of flight) sensor can be adopted.

[0251] Next, Figure 8G will be used to illustrate Figure 7C the extraction of the throat position in step S209. By superimposing on Figure 8F the reference numerals representing the respective parts of the user's body shown in Figure 10A , a double circle showing the throat position, and a black circle showing the chin position, the left image of Figure 8G is obtained.

[0252] The white area 211 corresponds to the throat 200 ( Figure 10A ), the very bright gray area 212 corresponds to the front part of the neck 201 ( Figure 10A ), and the bright gray area 213 corresponds to the root 202 of the jaw ( Figure 10A ). In addition, the medium gray area 214 corresponds to the chin 203 ( Figure 10A ), and the slightly darker gray area 215 corresponds to the part located in the face 204 ( Figure 10AThe lower part of the face corresponds to the lips and the lower part of the face around the lips. Furthermore, the darkest gray area 216 corresponds to the area located on face 204 ( Figure 10A The nose in the center corresponds to the upper part of the face around the nose.

[0253] Due to such Figure 10A As shown, the difference between distances Db and Dc is relatively small compared to the differences between other distances from the face direction detection window 13 to other parts of the user, so the difference between the reflected light intensity in the light gray area 213 and the medium gray area 214 is also small.

[0254] On the other hand, due to such Figure 10A As shown, among the distances from the detection window 13 in the facial direction to various parts of the user, distance Dn is the shortest, so the intensity of reflected light in the white area 211 corresponding to the throat 200 becomes the strongest.

[0255] Therefore, the overall control CPU (setting unit) 101 determines that region 211 corresponds to the throat 200 and its surroundings, and then sets the position 206, which is located in the center in the horizontal direction and closest to the camera / detection unit 10 (where it is located in the center of the throat 200 and its surroundings), to be the closest to the camera / detection unit 10. Figure 8G The position (represented by a double circle) is set as the center of head rotation (hereinafter referred to as throat position 206). The processing up to this point is... Figure 7C The content performed in step S209.

[0256] Next, we will use Figure 8G To explain Figure 7C The extraction of the chin position in step S210. Figure 8G In the image, a brighter medium gray area 214, which includes the chin, corresponds to the lower part of the face, which includes the lips, in the image 204. Figure 8G The curve on the right shows that the light intensity drops sharply in region 215, which is adjacent to region 214, because the rate of change of the distance relative to the face direction detection window 13 is larger.

[0257] The overall control CPU 101 determines that the brighter region 214 adjacent to the region 215 with a sharp drop in light intensity is the chin region. Furthermore, the overall control CPU 101 calculates (extracts) the position located at the center of region 214 in the lateral direction and furthest from the throat position 206 (by...). Figure 8G The black circle shown represents the chin position 207.

[0258] For example, Figure 8H and Figure 8I The changes are shown when the face is turned to the right. Figure 8H This shows how to make the user's face turn to the right by using... Figure 8E A graph of the difference image calculated using a similar method. Figure 8I The following graph shows the result, which is achieved by adjusting... Figure 8H The intensity of the difference image is scaled to fit the reflected light intensity of the infrared light projected onto the user's face and neck, and is obtained by superimposing a double circle showing the throat position 206 as the center of neck rotation and a black circle showing the chin position 207r.

[0259] Since the user's face is facing right, area 214 is located on the left side when viewed from below by the camera / detection unit 10. Figure 8I The area 214r shown moves. The area 215 corresponding to the lower part of the face, including the lips of the face 204, also moves to the left side of the area 215r when viewed from below by the camera / detection unit 10.

[0260] Therefore, the overall control CPU 101 determines that the brighter region 214r adjacent to the region 215r where the light intensity drops sharply is the region of the chin 203 (chin region). Furthermore, the overall control CPU 101 calculates (extracts) the position located at the center of the lateral direction of region 214r and furthest from the throat position 206 (by...). Figure 8I The black circle shown represents the chin position 207r.

[0261] Then, the control unit 101 calculates the movement angle θr, which represents the distance from the throat position 206 as the center. Figure 8G The image shows the chin position starting at 207 and moving to the right until... Figure 8I The chin is rotated and moved up to a position of 207r. For example... Figure 8I As shown, the movement angle θr is the angle by which the user's face moves in the lateral direction.

[0262] According to the above method, in step S210, the angle of the user's face in the horizontal direction (hereinafter referred to as the face angle) is calculated based on the chin position detected by the infrared detection device 27 of the face direction detection unit (three-dimensional detection sensor) 20.

[0263] Next, we will explain the detection of the upward-facing face. Figure 10B It is a diagram showing the user's face facing horizontally. Figure 10C This is a diagram showing the user's face tilted upwards at a 33° angle from the horizontal.

[0264] exist Figure 10B The distance from the face-oriented detection window 13 to the chin position 207 is Ffh, and... Figure 10C The distance from the face-oriented detection window 13 to the chin position 207u is Ffu. Since the chin position 207u moves upwards along with the face when the user faces upwards, therefore... Figure 10CAs shown, the distance Ffu becomes longer than the distance Ffh.

[0265] Figure 8J This is a diagram showing an image of a user with their face tilted upwards at a 33° angle from the horizontal, as seen from the face orientation detection window 13. Because... Figure 10C As shown, the user's face is turned upwards, so the face 204, including the lips and nose, is not visible from the facial direction detection window 13 located below the user's jaw. The chin 203 and its neck side are visible. Figure 8K Shown in Figure 10C The distribution of light intensity of reflected light 25 when the user is irradiated with infrared light 23 under the condition shown. Figure 8K The image on the left is a diagram showing the result as follows, which is achieved by adjusting the... Figure 8E The density of the difference image calculated using the same method is scaled to the light intensity of the reflected component of infrared light projected onto the user's face and neck, and is obtained by superimposing a double circle indicating the throat position 206 and a black circle indicating the chin position 207u. Figure 8K The two graphs in the image show the concentration changes in the left image. The left graph is equivalent to... Figure 8F The curve in the middle, and the curve on the right is equivalent to Figure 8G The curve in the graph.

[0266] By Figure 8F The reference numerals for the same light intensity regions shown are marked with a "u" to indicate this. Figure 8K The six regions corresponding to light intensity are 211u, 212u, 213u, 214u, 215u, and 216u. Although in Figure 8F The light intensity of the user's chin 203 is included in the mid-gray region 214, but... Figure 8K The light intensity shifts towards the black side and is included in the slightly darker gray region 215u. Thus, due to... Figure 10C As shown, the distance Ffu is longer than the distance Ffh, so the infrared detection device 27 can detect that the light intensity of the reflected light 25 from the chin 203 is reduced in a manner inversely proportional to the square of the distance.

[0267] Next, we will explain the detection of the downward-facing face. Figure 10D This is an example diagram showing a user's face tilted downwards at a 22° angle from the horizontal. Figure 10D In the middle, the distance from the face direction detection window 13 to the chin position 207d is Ffd.

[0268] Because when the user faces down, the chin position 207d moves downward along with the face, therefore... Figure 10D As shown, the distance Ffd becomes shorter than the distance Ffh, and the intensity of the reflected light 25 from the chin 203 becomes stronger.

[0269] Return to Figure 7C In step S211, the overall control CPU (distance calculation unit) 101 calculates the distance from the chin position to the face direction detection window 13 based on the light intensity detected by the infrared detection device 27 of the face direction detection unit (three-dimensional detection sensor) 20. At this time, the face angle in the vertical direction is also calculated.

[0270] For example, when Figure 10B As shown, when the user is facing forward, the relationship between the vertical angle θh of the line connecting the chin position 207 and the face direction detection window 13 relative to the horizontal direction and the horizontal component Lh of the distance Ffh from the face direction detection window 13 to the chin position 207 is represented by Lh = Ffh * cosθh.

[0271] Furthermore, when such Figure 10C As shown, when the user faces upward at 33°, the relationship between the vertical angle θu of the line connecting the chin position 207u and the face direction detection window 13 and the horizontal component Lu of the distance Ffu from the face direction detection window 13 to the chin position 207u is represented by Lu=Ffu*cosθu.

[0272] Therefore, the overall control CPU 101 calculates the difference between θh and θu as the facial angle in the vertical direction. It should be noted that the distances Ffh and Ffu are respectively based on... Figure 8G The light intensity at the chin position 207 and Figure 8K The light intensity at the chin position (207µm) is used for calculation.

[0273] Furthermore, the horizontal components Lh and Lu are respectively transmitted through... Figure 8G The distance Ffh from the chin position 207 in the facial image and to Figure 8K The distance Ffu from the chin position (207u) in the facial image is converted into the actual size of the subject for calculation.

[0274] It should be noted that the angle calculation described above presupposes that the distance from the throat position 206, which serves as the center of head rotation, to the chin position 207 is almost equal to the distance from the face-direction detection window 13 to the chin position 207. The calculation method becomes more complex when the camera / detection unit 10 is installed in a separate position.

[0275] In step S212, the overall control CPU 101 stores the facial angles obtained in step S210 in the lateral direction (first detection direction) and the facial angles obtained in step S211 in the vertical direction (second detection direction) as the user's three-dimensional observation direction vi ("i" is an arbitrary reference numeral) in the main memory 103.

[0276] Although the facial angle in the vertical direction is calculated in step S211 by detecting the distance relative to the facial direction detection window 13, the facial angle can be calculated by another method. For example, the change in facial angle can be calculated by comparing the level of change in light intensity at the chin 203. That is, the change can be calculated by... Figure 8G The gradient CDh of reflected light intensity from the root of the jaw 202 to the chin 203 in the curve diagram is compared with... Figure 8K The change in facial angle is calculated by comparing the gradient CDu of reflected light intensity from the root of the jaw 202 to the chin 203 in the curve diagram.

[0277] Figure 7D It is shown Figure 7A The flowchart for the recording direction / region determination subroutine in step S300 is shown. Before explaining the details of this process, we will first use... Figure 11A This embodiment uses an ultra-wide-angle image as the object for determining the recording direction and recording area.

[0278] In the camera body 1 of this embodiment, the imaging unit 40 uses an ultra-wide-angle camera lens 16 to capture an ultra-wide-angle image of the periphery of the imaging / detection unit 10. An image of the observation direction can be obtained by extracting a portion of this ultra-wide-angle image.

[0279] Figure 11A This is a diagram showing the target field of view 125 set in an ultra-wide-angle image captured by the camera unit 40 with the user facing forward.

[0280] like Figure 11A As shown, the pixel region 121 that can be captured by the solid-state image sensor 42 is a rectangular region. Furthermore, the effective projection region (predetermined region) 122 is the region of a circular hemispherical image of a fisheye image projected onto the solid-state image sensor 42 by the camera lens 16. The camera lens 16 is adjusted so that the center of the pixel region 121 coincides with the center of the effective projection region 122.

[0281] The outermost periphery of the circular effective projection area 122 indicates a position with a field of view (FOV) angle of 180°. When the user is looking at the center of both the vertical and horizontal directions, the angular range of the captured and recorded target field of view 125 becomes 90° (half the FOV angle) centered on the center of the effective projection area 122. It should be noted that the camera lens 16 of this embodiment can also introduce light outside the effective projection area 122 and can project light within the maximum FOV angle of 192° onto the solid-state image sensor 42 as a fisheye image. However, optical performance degrades significantly in the area outside the effective projection area 122. For example, resolution drops sharply, light intensity decreases, and distortion increases. Therefore, in this embodiment, the image in the viewing direction is extracted only from the image projected onto the pixel area 121 within the hemispherical image displayed on the effective projection area 122 (hereinafter referred to as the ultra-wide-angle image) as the recording area.

[0282] Because the effective projection area 122 is larger in the vertical direction than the shorter side of the pixel area 121, in this embodiment, the upper and lower edges of the image in the effective projection area 122 are outside the pixel area 121. However, the relationship between the various areas is not limited to this. For example, the optical system can be designed such that by changing the configuration of the camera lens 16, the entire effective projection area 122 will be included in the pixel area 121. The invalid pixel area 123 is the portion of the pixel area 121 that is not included in the effective projection area 122.

[0283] The target field of view 125 shows the region of the image extracted from the ultra-wide-angle image in the user's viewing direction. The target field of view 125 is defined by the field angles (45° in this case, 90° FOV) centered on the viewing direction. Figure 11A In the example, since the user is facing forward, the center of the target field of view 125 becomes the observation direction (center direction) vo that is consistent with the center of the effective projection area 122.

[0284] Figure 11A The ultra-wide-angle image shown includes subject A 131 as a child, subject B 132 showing the child as subject A attempting to climb the steps, and subject C 133 as a motorcycle-shaped amusement park ride.

[0285] Next, in Figure 7D The diagram shows how to use Figure 11A The ultra-wide-angle image is obtained by performing the following operation: Figure 7A The recording direction / area determination process in step S300. Below, a specific example illustrating the target field of view 125 is used. Figures 12A to 12G This will explain the process.

[0286] In step S301, the preset field angle setting value V is obtained by reading from the main memory 103. ang .

[0287] In this embodiment, the internal non-volatile memory 102 stores all available field angles (45°, 90°, 110°, and 130°) as field angle setting values ​​V. ang The image extraction / development unit 50 extracts the image from the ultra-wide-angle image based on the field angle setting value V. ang An image of the viewing direction within the defined region. Furthermore, an image was created within... Figure 7B The field angle setting value V included in the various setting values ​​read from the internal non-volatile memory 102 in one of steps S103, S106, and S108. ang And store it in main memory 103.

[0288] Furthermore, in step S301, the observation direction vi determined in step S212 is set as the recording direction, the image in the target field of view 125 is extracted from the ultra-wide-angle image, and the extracted image is stored in the main memory 103. The center of the target field of view 125 is specified by the observation direction vi, and the area of ​​the target field of view 125 is determined by the obtained field angle setting value V. ang To limit.

[0289] In the case of the observation direction (center direction) vo, since the effect of optical distortion caused by the camera lens 16 is mostly ignored, the shape of the established target field of view 125 is the same as that of the target field of view 125o after distortion conversion in step S303 described later. Figure 12A The shapes are almost identical. In the following text, the target field of view after distortion conversion in the observation direction vi is referred to as the target field of view 125i.

[0290] Next, in step S302, a preset image stabilization level is obtained by reading from the main memory 103.

[0291] In this embodiment, as described above, the image stabilization level included in the various setting values ​​read from the internal non-volatile memory 102 in one of steps S103, S106 and S108 is established and stored in the main memory 103.

[0292] Furthermore, in step S302, the number of image stabilization margin pixels P is set based on the obtained image stabilization level. is .

[0293] In image stabilization processing, an image following in the opposite direction to the blur direction is obtained based on the amount of blur in the camera / detection unit 10. Therefore, in this embodiment, an image stabilization margin required for image stabilization is established around the target field of view 125i.

[0294] Furthermore, in this embodiment, the internal non-volatile memory 102 stores the number of image stabilization margin pixels P. is The values ​​are maintained in a table that are associated with the corresponding image stabilization level. For example, in the case of an image stabilization level of "medium", the width established around the target field of view is the number of image stabilization margin pixels P read from the table above. is The image stabilization margin of "100 pixels".

[0295] Figure 12E It is shown in Figure 12A The diagram illustrates an example of an image stabilization margin corresponding to a predetermined image stabilization level around a target field of view of 125°. The diagram will explain an image stabilization level of "medium" (i.e., an image stabilization margin of P pixels). is (This refers to the case of "100 pixels").

[0296] like Figure 12E As shown by the dotted lines, the width P is established on the top, bottom, left, and right sides of the target field of view of 125° as the image stabilization margin. is The image stabilization margin for "100 pixels" is 126°.

[0297] Figure 12A and Figure 12E The example shown illustrates the case where the observation direction vi coincides with the center O (optical axis center of the camera lens 16) of the effective projection area 122 (observation direction vo) for simplification. On the other hand, when the observation direction vi points to the periphery of the effective projection area 122, a conversion is needed to mitigate the effects of optical distortion.

[0298] In step S303, the shape of the target field of view 125 established in step S301 is corrected (distortion is converted) taking into account the observation direction vi and the optical properties of the camera lens 16 to generate the target field of view 125i. Similarly, the number of image stabilization margin pixels P set in step S302 is also corrected taking into account the observation direction vi and the optical properties of the camera lens 16. is .

[0299] For example, the field angle setting value V angThe angle should be 90°, and the user should observe a 45° rightward direction starting from the center o. In this case, the observation direction vr (vector information [45°, 0°]) is determined in step S212, and the area 45° to the left and right and 45° up and down centered on the observation direction vr becomes the target field of view 125. Furthermore, taking into account the optical properties of the camera lens 16, the target field of view 125 is corrected to... Figure 12B The target field of view shown is 125r.

[0300] like Figure 12B As shown, the target field of view 125r becomes wider towards the periphery of the effective projection area 122. Furthermore, the position of the observation direction vr is slightly closer to the inner side relative to the center of the target field of view 125r. This is because the optical design of the camera lens 16 in this embodiment is similar to that of a stereoscopic projection fisheye lens. It should be noted that the correction depends on the optical design of the camera lens 16. If the camera lens 16 is designed as an equidistant projection fisheye lens, an equal solid angle projection fisheye lens, or an orthographic projection fisheye lens, the target field of view 125 is corrected according to its optical properties.

[0301] Figure 12F It is shown in Figure 12B The target field of view shown is 125r around which is assigned Figure 12E The figure shows an example of an image stability margin of 126r corresponding to the same image stability level "medium".

[0302] The width is the number of pixels P used as an image stability margin. is An image stabilization margin of 126° was established on the left, right, top, and bottom sides of the target field of view of "100 pixels". Figure 12E In comparison, the image stabilization margin is 126r. Figure 12F Image stabilization margin (Number of pixels P) is Increase towards the periphery of the effective projection area 122.

[0303] In this way, the shape of the image stabilization margin established around the target field of view 125r is also corrected, just like the shape of the target field of view 125r, so that... Figure 12F As shown in the image stabilization margin 126r, the correction amount will increase towards the periphery of the effective projection area 122. This is also because the optical design of the camera lens 16 in this embodiment is close to the optical design of a stereoscopic projection fisheye lens. It should be noted that the correction depends on the optical design of the camera lens 16. If the camera lens 16 is designed as an equidistant projection fisheye lens, an equal solid angle projection fisheye lens, or an orthographic projection fisheye lens, the image stabilization margin 126r is corrected according to its optical properties.

[0304] The process of sequentially switching the shape of the target field of view 125 and its image stabilization margin, taking into account the optical properties of the camera lens 16, performed in step S303 is complex. Therefore, in this embodiment, the process of step S303 is performed using a table stored in the internal non-volatile memory 102 that maintains the shape of the target field of view 125i and its image stabilization margin for each viewing direction vi. It should be noted that the overall control CPU 101 may have a calculation formula depending on the optical design of the camera lens 16. In this case, the overall control CPU 101 can use this calculation formula to calculate the optical distortion value.

[0305] In step S304, the position and size of the image recording frame are calculated. As described above, an image stabilization margin 126i is established around the target field of view 125i. However, when the position of the viewing direction vi is close to the periphery of the effective projection area 122, for example, as shown by the image stabilization margin 126r, the shape of the image stabilization margin becomes quite special.

[0306] The overall control CPU 101 can extract images only from areas of this specific shape and apply display processing to the extracted images. However, non-rectangular images are generally not used when recording as image data in step S600 or when sending image data to the display device 800 in step S700. Therefore, in step S304, the position and size of the rectangular image recording frame 127i, which includes the entire image stability margin 126i, are calculated.

[0307] Figure 12F The image recording frame 127r calculated in step S304 for the image stability margin 126r is shown using a dashed line.

[0308] In step S305, the position and size of the image recording frame 127i calculated in step S304 are recorded in the main memory 103.

[0309] In this embodiment, the top-left coordinates (Xi, Yi) of the image recording frame 127i in the ultra-wide-angle image are recorded as the position of the image recording frame 127i, and the horizontal width WXi and vertical width WYi starting from coordinates (Xi, Yi) are recorded as the size of the image recording frame 127i. For example, in step S305, the following is recorded: Figure 12F The image recording frame 127r shown has coordinates (Xr, Yr), a horizontal width WXr, and a vertical width WYr. It should be noted that the coordinates (Xi, Yi) are XY coordinates with the origin at a predetermined reference point (specifically, the optical center of the camera lens 16).

[0310] With the image stabilization margin 126i and the image recording frame 127i thus determined, processing from... Figure 7DThe subroutine shown exits.

[0311] The explanations so far have described viewing directions with a horizontal angle of 0° (such as viewing direction v0 (vector information [0°, 0°]) and viewing direction vr (vector information [45°, 0°]), etc.) to simplify the explanation of complex optical distortion conversion. On the other hand, the user's actual viewing direction vi is arbitrary. Therefore, the following will explain the imaging processing of the recording area when the horizontal angle is not 0°. For example, the field angle setting value V ang When the angle is 90° and the viewing direction vm is [-42°, -40°], such as Figure 12C As shown, the target's field of view is 125m.

[0312] Furthermore, even when the observation direction vm (vector information [-42°, -40°]) is the same as the target field of view of 125m, the field angle setting value V ang At a 45° angle, such as Figure 12D As shown, a target field of view of 128m appears, which is slightly smaller than the target field of view of 125m.

[0313] In addition, such as Figure 12G As shown, an image stabilization margin of 129m and an image recording frame of 130m are established around the target field of view of 128m.

[0314] Since the processing in step S400 is a basic camera operation and employs the general sequence of the camera unit 40, its detailed description is omitted. It should be noted that the image signal processing circuit 43 in the camera unit 40 of this embodiment also performs processing to convert the signal from the inherent output format (standard examples: MIPI, SLVS) of the solid-state image sensor 42 into camera data for a general sensor reading system.

[0315] When the video image mode is selected via the camera mode switch 12, the camera unit 40 begins recording in response to the pressing of the start switch 14. Recording then ends when the stop switch 15 is pressed. Conversely, when the still image mode is selected via the camera mode switch 12, the camera unit 40 captures a still image each time the start switch 14 is pressed.

[0316] Figure 7E It is shown Figure 7A The flowchart of the subroutine for recording area imaging processing in step S500.

[0317] In step S501, the raw data of the entire area of ​​the camera data (ultra-wide-angle image) generated by the camera unit 40 in step S400 is obtained, and the raw data is input into the camera unit of the head unit (not shown) called the overall control CPU 101.

[0318] Next, in step S502, based on the coordinates (Xi, Yi), horizontal width WXi, and vertical width WYi recorded in the main memory 103 in step S305, the image within the image recording frame 127i is extracted from the ultra-wide-angle image obtained in step S501. After this extraction, cropping and image processing consisting of steps S503 to S508 is performed only on pixels within the image stabilization margin 126i. Figure 7F This significantly reduces computation compared to performing image processing on the entire area of ​​the ultra-wide-angle image read in step S501. Therefore, computation time and power consumption can be reduced.

[0319] like Figure 7F As shown, when the video image mode is selected via the camera mode switch 12, the processing steps S200 and S300, as well as the processing step S400, are executed in parallel at the same or different frame rates. Whenever the raw data of the entire area of ​​a frame generated by the camera unit 40 is obtained, cropping and display processing is performed based on the coordinates (Xi, Yi), horizontal width WXi, and vertical width WYi recorded in the main memory 103 at that time point.

[0320] When cropping and image processing begins with pixels within an image stabilization margin of 126i, color interpolation is performed in step S503 to interpolate the data of the color pixels arranged in Bayer array. Then, white balance is adjusted in step S504, and color conversion is performed in step S505. In step S506, gamma correction is performed to correct the grayscale based on a preset gamma correction value. In step S507, edge enhancement is performed according to the image size.

[0321] In step S508, the image data is converted into a data format that can be primarily stored by applying processing such as compression. The converted image data is stored in main memory 103. Afterward, the process exits from this subroutine. Details of the primarily storable data formats will be explained later.

[0322] The order and presence of the processes in steps S503 to S508 performed during the cropping and developing process can be varied depending on the nature of the camera system and are not limiting to the present invention. Furthermore, when a video image mode is selected, the processes in steps S200 to S500 are repeated until recording ends.

[0323] According to this process, the computational load is significantly reduced compared to performing imaging processing on the entire area read in step S501. Therefore, an inexpensive and low-power microcomputer can be used as the overall control CPU 101. Furthermore, heat generation in the overall control CPU 101 is reduced, and battery consumption 94 is also reduced.

[0324] Furthermore, in order to reduce the control load on the overall control CPU 101, in this embodiment, optical correction processing of the image is performed ( Figure 7A Step S800) and image stabilization processing ( Figure 7A Step S900 is not performed by the camera body 1. These processes are performed by the display device controller 801 after the image is transmitted to the display device 800. The data extracted from the image does not include position information substituted into formulas in the optical correction process or referenced from a correction table in the image stabilization process. Therefore, if only data from a portion of the image extracted from the projected ultra-wide-angle image is transmitted to the display device 800, the optical correction process and image stabilization process cannot be performed correctly. Therefore, in this embodiment, the camera body 1 sends correction data, including information related to the extraction position of the image from the ultra-wide-angle image, to the display device 800 along with the data of the extracted image.

[0325] When the extracted image is a still image, since the still image data and correction data correspond one-to-one, the display device 800 can correctly perform optical correction and image stabilization processing even if these data are sent separately. On the other hand, when the extracted image is a video image, if the video image data and correction data are sent separately to the display device 800, it becomes difficult to determine the correspondence between each frame of the video image data and the correction data. Specifically, if the clock rate of the overall control CPU 101 in the camera body 1 is slightly different from the clock rate of the display device controller 801 in the display device 800, synchronization between the overall control CPU 101 and the display device controller 801 will be lost during several minutes of video image capture operations. This may result in the display device controller 801 using correction data that differs from the corresponding correction data to correct frames, leading to defects.

[0326] Therefore, in this embodiment, when the extracted video image data is sent to the display device 800, the camera body 1 appropriately assigns its correction data to the video image data. The method is described below.

[0327] Figure 14 It is shown Figure 7A The flowchart of the main record processing subroutine in step S600 is shown below. (See also the reference below.) Figure 15 This will explain the process. Figure 14 The process is illustrated when a video image mode is selected via camera mode switch 12. When a still image mode is selected, the process begins at step S601 and ends after the process in step S606.

[0328] In step S601a, the overall control CPU 101 processes the image in the recording area ( Figure 7E The system reads an image of a frame from the video image after it has been displayed, a frame for which the processing steps S601 to S606 were not applied. Furthermore, the overall control CPU (metadata generation unit) 101 generates correction data as metadata for the read frame.

[0329] In step S601, the overall control CPU 101 appends information related to the extraction position of the image of the frame read in step S601a to the correction data. The information appended in this step is the coordinates (Xi, Yi) of the image recording frame 127i obtained in step S305. It should be noted that the information appended in this step can be vector information indicating the viewing direction vi.

[0330] In step S602, the overall control CPU (optical correction value acquisition unit) 101 acquires the optical correction value. The optical correction value is the optical distortion value set in step S303. Optionally, the optical correction value can be a correction value corresponding to the optical properties of the lens, such as an edge light amount correction value or a diffraction correction value.

[0331] In step S603, the overall control CPU 101 appends the optical correction value used in the distortion conversion in step S602 to the correction data.

[0332] In step S604, the overall control CPU 101 determines whether the image stabilization mode is effective. Specifically, if the preset image stabilization mode is "medium" or "strong," it determines that the image stabilization mode is effective and proceeds to step S605. On the other hand, if the preset image stabilization mode is "off," it determines that the image stabilization mode is invalid and proceeds to step S606. The reason for skipping step S605 when the image stabilization mode is "off" is that by skipping step S605, the amount of computational data of the overall control CPU 101 and the amount of wireless communication data are reduced, and the power consumption and heat generation of the camera body 1 can be reduced. Although the reduction of data used in image stabilization processing is explained, the data related to edge light amount correction values ​​or diffraction correction values ​​obtained as optical correction values ​​in step S602 can also be reduced.

[0333] In this embodiment, although the image stabilization mode is preset by the user's operation of the display device 800, it can be set as the default setting of the camera body 1. Furthermore, if the camera system is configured to switch the effectiveness of image stabilization processing after transmitting image data to the display device 800, the process can skip step S604 and proceed directly from step S603 to step S605.

[0334] In step S605, the overall control CPU (motion detection unit) 101 appends the image stabilization mode obtained in step S302 and the gyroscope data associated with the frame read and stored in the main memory 813 in step S601a to the correction data.

[0335] In step S606, the overall control CPU 101 updates the video file 1000 using data obtained by encoding the image data of the frame read in step S601a and the correction data with various data appended in steps S601 to S605. Figure 15 It should be noted that, in the case of reading the first frame of the video image in step S601a, video file 1000 is generated in step S606.

[0336] In step S607, the overall control CPU 101 determines whether the image processing through the recording area has been read ( Figure 7E All frames of the displayed video image are read. If not all frames are read, the process returns to step S601a. ​​On the other hand, if all frames are read, the process exits from the subroutine. The generated video file 1000 is stored in the internal non-volatile memory 102. In addition to the main memory 813 and the internal non-volatile memory 102, the video file can also be stored in the mass non-volatile memory 51. Furthermore, a transmission process is performed to immediately transmit the generated image file 1000 to the display device 800. Figure 7A (Step S700 in the process). The image file 1000 can be stored in the main memory 813 after it has been transferred to the display device 800.

[0337] In this embodiment, encoding means combining image data and correction data into a single file. At this point, the image data can be compressed, or the data file composed of the image data and correction data can be compressed.

[0338] Figure 15 This is a diagram illustrating the data structure of video file 1000. Video file 1000 consists of a header portion 1001 and a frame portion 1002. Frame portion 1002 consists of frame datasets, each frame dataset consisting of the image of each frame and the corresponding frame metadata. That is, frame portion 1002 includes a frame dataset representing the total number of frames of video images.

[0339] In this embodiment, the frame metadata is information obtained by encoding correction data, which includes, if necessary, the extraction location (position information within the image), optical correction values, and gyroscope data. However, the frame metadata is not limited to this. The amount of information in the frame metadata can be changed. For example, additional information can be added to the frame metadata according to the camera mode selected via camera mode switch 12. Alternatively, a portion of the information in the frame metadata can be deleted.

[0340] The header section 1001 records offset values ​​up to the frame dataset of each frame or the start address of each frame. Optionally, metadata such as the time and size corresponding to the video file 1000 may be stored in the header section 1001.

[0341] Thus, in the main record processing ( Figure 14 In the process, video file 1000 is transmitted to display device 800. Video file 100 includes the following datasets, each of which is processed by imaging through the recording area (...). Figure 7E The video image displayed is composed of frames and their metadata. Therefore, even if the clock frequency of the overall control CPU 101 in the camera body 1 is slightly different from the clock frequency of the display device controller 801 in the display device 800, the display device controller 801 appropriately applies correction processing to the video image displayed in the camera body 1.

[0342] In this embodiment, although optical correction values ​​are included in the frame metadata, optical correction values ​​can be applied to the entire video image.

[0343] Figure 16 It is shown in Figure 7A The flowchart shows the subroutine for sending data to the display device 800, which is executed in step S700. Figure 16 The process is shown when a video image mode is selected via camera mode switch 12. It should be noted that when a still image mode is selected, the process begins from step S702.

[0344] In step S701, it is determined whether the video image capture processing using the camera unit 40 (step S400) has ended or is still being recorded. If the video image is being recorded, the image file 1000 in the recording area display processing for each frame (step S500) and the main recording processing (step S600) is updated sequentially (step S606). Since wireless transmission has a high power load, if wireless transmission is performed in parallel during video image capture, the battery 94 needs to have a large battery capacity or new measures to address heat generation. Furthermore, from a computing power perspective, if wireless transmission is performed in parallel during video image capture, the computing load will increase, requiring a high-specification CPU as the overall control CPU 101, thereby increasing costs. Considering these points, in this embodiment, after the video image capture operation ends ("Yes" in step S701), the overall control CPU 101 causes the processing to proceed to step S702 and establishes a wireless connection with the display device 800. On the other hand, if the camera system of this embodiment has a margin in terms of power supplied from the battery 94 and does not require new measures to address heat generation, the overall control CPU 101 can pre-establish a wireless connection with the display device 800 when the camera body 1 is started or before recording begins.

[0345] In step S702, the overall control CPU 101 establishes a connection with the display device 800 via the high-speed wireless communication unit 72 to transmit the large video file 1000 to the display device 800. It should be noted that the low-power wireless communication unit 71 is used to transmit low-resolution images for checking field angles to the display device 800 and to exchange various setting values ​​with the display device 800. On the other hand, the low-power wireless communication unit 71 is not used for transmitting the video file 1000 due to the longer transmission time.

[0346] In step S703, the overall control CPU 101 transmits the video file 1000 to the display device 800 via the high-speed wireless communication unit 72. Upon completion of the transmission, the overall control CPU 101 initiates the process to step S704. In step S704, the overall control CPU 101 closes the connection with the display device 800 and exits the subroutine.

[0347] So far, we have described the case where an image file comprises all frames of a single video image. On the other hand, if the recording time of the video image is longer than several minutes, the video image can be divided into multiple image files on a unit-time basis. In a video file with... Figure 15 With the data structure shown, even if a video image is transmitted to the display device 800 as multiple image files, the display device 800 can correct the video image without any timing gap with the correction data.

[0348] Figure 17 It is shown Figure 7A The flowchart of the optical correction processing subroutine in step S800 is shown below. Also referenced below... Figures 18A to 18F To illustrate this process, as described above, this process is executed by the display device controller 801 of the display device 800.

[0349] In step S801, the display device controller (video file receiving unit) 801 first receives the video file 1000 transmitted from the camera body 1 during the transmission process to the display device 800 (step S700). Then, the display device controller (first extraction unit) 801 obtains the optical correction values ​​extracted from the received video file 1000.

[0350] In the next step S802, the display device controller (second extraction unit) 801 obtains an image (an image of a frame of the video image) from the video file 1000.

[0351] In step S803, the display device controller (frame image correction unit) 801 uses the optical correction value obtained in step S801 to correct the optical aberration of the image obtained in step S802, and stores the corrected image in the main memory 813. If the image obtained in step S802 is extracted during optical correction, an image region narrower than the display area (target field of view 125i) determined in step S303 is extracted (extracted display area), and this processing is performed on that image region.

[0352] Figures 18A to 18F It is used for explanation Figure 17 The diagram shows the application of distortion correction in step S803.

[0353] Figure 18A This is a diagram showing the position of the subject 1401 as seen by the user with the naked eye when taking an image. Figure 18B This is a diagram showing an image of the subject 1401 formed on the solid-state image sensor 42.

[0354] Figure 18C It is shown Figure 18B The image shows the display area 1402 in the image. The display area 1402 is the extracted display area described above.

[0355] Figure 18D This is a diagram showing the extracted image obtained by extracting the image of the imaging region 1402. Figure 18E It shows that by... Figure 18D The image shown is the result of correcting distortions extracted from the developed image. Because the extraction process is performed during the correction of distortions in the developed image, therefore... Figure 18E The field angle of the image shown becomes greater than Figure 18D The field angle of the extracted and developed image shown is smaller.

[0356] Figure 19 It is shown Figure 7A The flowchart of the image stabilization subroutine in step S900 is shown below. This process will be described with reference to FIG25. As described above, this process is performed by the display device controller 801 of the display device 800.

[0357] In step S901, the display device controller 801 obtains the blur amount Vo and gyroscope data of the currently processed frame (current frame) and the blur amount V of the frame that is the immediate preceding frame (previous frame) from the frame metadata of the video file 1000. n-1 And gyroscope data. Then, based on this information, a coarse ambiguity quantity V is calculated. n Pre .

[0358] In step S902, the display device controller 801 calculates the fine blur amount V based on the video file. n Det Blur is detected by calculating the amount of movement of feature points in the image from the previous frame to the current frame.

[0359] Feature points can be extracted using known methods. For example, a method using a luminance information image generated by extracting only the luminance information of the image frame can be employed. This method subtracts an image from the original luminance information image that shifts the original luminance information image by one or more pixels. Pixels whose absolute difference exceeds a threshold are extracted as feature points. Furthermore, edges can be extracted as feature points by subtracting an image generated by applying a high-pass filter to the original luminance information image.

[0360] When shifting the brightness information images of the current frame and the previous frame by one or more pixels, the difference is calculated multiple times. The shift amount is obtained by calculating the position where the difference decreases at the pixel of the feature point.

[0361] Since multiple feature points are required as described later, it is preferable to divide each image of the current frame and previous frames into multiple blocks, and extract feature points for each block. The block segmentation depends on the number of pixels and the aspect ratio of the image. Generally, 12 blocks of 4x3 or 54 blocks of 9x6 are preferred. If the number of blocks is too small, trapezoidal distortion and rotational blur around the optical axis caused by the tilt of the imaging unit 40 of the camera body 1 cannot be correctly corrected. On the other hand, if the number of blocks is too large, the size of each block becomes smaller, thus shortening the distance between adjacent feature points, which leads to errors. Therefore, the optimal number of blocks is selected based on factors such as the number of pixels, the ease of feature point detection, and the field angle of the subject.

[0362] Because calculating the shift amount requires multiple difference calculations when shifting the brightness information images of the current frame and the previous frame by one or several pixels, the computational complexity increases. This is because the shift amount is actually based on a coarse blur amount V. n Pre It is calculated based on the deviation (number of pixels) from it, so the difference calculation is only performed near the coarse blur amount, which can significantly reduce the amount of computation.

[0363] Next, in step S903, the display device controller 801 uses the fine fuzzy quantity V obtained in step S902. n Det Then, image stabilization is performed. The process then exits from this subroutine.

[0364] It should be noted that as methods for image stabilization, Euclidean and affine transformations that allow rotation and parallel translation, and projection transformations that allow trapezoidal correction are known.

[0365] Although Euclidean transformation can correct for movement and rotation in the X and Y axes, it cannot correct for blur caused by camera shake of the camera unit 40 of the camera body 1 in the forward / backward or pan / tilt directions. Therefore, in this embodiment, an affine transformation capable of correcting skew is used to perform image stabilization. The affine transformation from the coordinates (x, y) of the feature point used as a standard to the coordinates (x', y') is represented by the following Equation 1.

[0366]

[0367] If at least three feature points are detected as deviations, the affine coefficients of the 3x3 matrix in Formula 1 can be calculated. However, when the detected feature points are close to each other or aligned on a straight line, image stabilization becomes inaccurate in regions far from the feature points or the straight line. Therefore, it is preferable to select feature points that are far apart from each other and not on a straight line. Thus, when multiple feature points are detected, feature points that are close to each other are excluded, and the remaining feature points are standardized using the least squares method.

[0368] Figure 18F This illustrates applying the image stabilization process of step S903. Figure 18E The image shown is the result of the distortion-corrected image. Because extraction processing was performed during image stabilization, therefore... Figure 18F The field angle of the image shown becomes smaller than Figure 18E The field angle of the image shown.

[0369] It is possible to obtain a high-quality image that has been corrected for blur by performing such image stabilization processing.

[0370] The above describes a series of operations performed by the camera body 1 and the display device 800 included in the camera system of this embodiment.

[0371] When the user selects the video image mode via the camera mode switch 12 after turning on the power switch 11, and observes the front without rotating their face vertically or horizontally, the face orientation detection unit 20 detects... Figure 12A The observation direction vo (vector information [0°, 0°]) is shown. Then, the recording direction / field angle determination unit 30 extracts the direction from the ultra-wide-angle image projected onto the solid-state image sensor 42. Figure 12A The image shown is in the target field of view 125°. Figure 11B ).

[0372] Then, when the user begins to observe, for example, without operating the camera body 1... Figure 11A When the child (subject A 131) is detected, the facial orientation detection unit 20 detects as follows: Figure 11C The observation direction vm is shown (vector information [-42°, -40°]). Then, the recording direction / field angle determination unit 30 extracts the image of the target field of view 125m from the ultra-wide-angle image captured by the camera unit 40. Figure 11C ).

[0373] Thus, in steps S800 and S900, the display device 800 applies optical correction processing and image stabilization processing to the image of the shape extracted according to the viewing direction. Therefore, even if the overall control CPU 101 of the camera body 1 has low specifications, the target field of view will be 125m (…). Figure 11C The obviously distorted image in ) is converted into such Figure 11D The image shown is a child (subject A) 131 centered on a corrected blur and distortion image. That is, even if the user does not touch the camera body 1 except to turn on the power switch 11 and select the mode using the camera mode switch 12, the user can obtain an image taken in their own viewing direction.

[0374] In this embodiment, although the face direction detection unit 20 and the imaging unit 40 are described as being integrally formed in the camera body 1, the configuration is not limited to this. The face direction detection unit 20 can be worn on a part of the user's body other than the head, and the imaging unit 40 can be worn on the user's body. For example, in this embodiment, the imaging unit 40 can be worn on the shoulder or abdomen. However, when the imaging unit 40 is worn on the right shoulder, the subject on the left side is obscured by the head. In this case, it is preferable to wear multiple imaging units, including the right shoulder. Furthermore, when the imaging unit 40 is worn on the abdomen, spatial parallax occurs between the imaging unit 40 and the head. In this case, it is preferable to perform a correction calculation for the observation direction to compensate for this parallax.

[0375] The second embodiment will be described below. In the second embodiment, using Figures 20A to 23E This section details the methods for calibrating individual differences among users wearing the camera body 1 and adjusting for those differences.

[0376] This embodiment will be described essentially as a derivative of the first embodiment. Therefore, structures in the camera system of the second embodiment that are identical in structure to those in the camera system of the first embodiment are indicated by the same reference numerals, and repeated descriptions are omitted. Different structures will be described with added details.

[0377] Users wearing the camera body 1 have individual differences and adjustment variations, such as physique, the tilt angle around the neck when wearing the camera body 1, the state of clothing such as the collar, and the adjustment state of the straps 82L and 82R. Therefore, the optical axis center of the camera lens 16 of the camera body 1 is usually not aligned with the center of the field of view when the user is facing forward (hereinafter referred to as the user's natural state). For the user, it is preferable to align the center of the recording area (target field of view 125) with the center of the user's field of view in the current posture or operation, rather than aligning the center of the recording area with the optical axis center of the camera lens 16 of the camera body 1.

[0378] Furthermore, individual differences exist not only in the user's field of view center in a natural state, but also in the field of view center depending on the head orientation (up, down, right, left, or tilted) and in the head's movement space. Therefore, individual differences also arise in the relationship between the facial orientation (observation direction) detected by the facial orientation detection unit 20 and the center position of the target field of view 125 established based on the observation direction (hereinafter referred to as the field of view center position). Therefore, a calibration operation that associates the facial orientation with the field of view center position is required.

[0379] Typically, calibration is preferred as a Figure 7A This is performed as part of the preparation process (step S100). It is assumed that the calibration operation is typically performed upon the first startup of the camera body 1. It should be noted that calibration can also be performed if a predetermined time has elapsed since a previous calibration, or if the position of the camera body 1 relative to the user changes from its previous calibration position. Calibration can also be performed if the face orientation detection unit 20 can no longer detect the user's face. Furthermore, calibration can be performed when the user puts the camera body 1 back on, if the user is detected removing the camera body 1. Therefore, it is preferable to perform the calibration operation appropriately at the timing when it is determined that calibration is needed for proper use of the camera body 1.

[0380] Figure 20A and Figure 20B This is a diagram illustrating details of the calibrator 850 used in the calibration process according to the second embodiment. In this embodiment, the calibrator 850 is combined with the functionality of the display device 800.

[0381] In addition to including as Figure 1D In addition to the components of the display device 800 shown, including button 802, display unit 803, built-in camera 805, face sensor 806, and angular velocity sensor 807, it also includes a positioning indicator 851 and a calibration button 854. Figure 20A The B button 804 provided in the first embodiment is not shown in the figure because the B button 804 is not used in this embodiment and can be replaced by the calibration button 854 as described later.

[0382] Figure 20A This illustrates the case where the positioning indicator 851 is a specific pattern displayed on the display unit 803. Figure 20B This illustrates a scenario where the appearance of the calibrator 850 is used as a positioning indicator. Figure 20B In this case, the positioning index center 852, as described later, is calculated based on information related to the contour of the calibrator 850.

[0383] It should be noted that positioning indicators are not limited to Figure 20A and Figure 20B Examples. For instance, the positioning indicator can be separate from the calibrator 850. The positioning indicator can be anything, as long as its size is easily measurable and its shape is suitable for the user's viewing. For example, the positioning indicator could be the lens cap of the camera lens 16 or the charging unit used by the camera body 1. In any case, since the basic mindset in calibration operations is common, the following examples and explanations will primarily illustrate this. Figure 20A The calibrator shown is 850.

[0384] It should be noted that the calibrator 850 in this embodiment will be combined with the functions of the display device 800. Furthermore, the calibrator 850 may be, for example, a dedicated device, a general-purpose smartphone, or a tablet terminal.

[0385] The positioning indicator 851 is displayed on the display unit 803 of the calibrator 850 and is a graphic representation of the horizontal width L851a, vertical width L851b, and center 852 of the positioning indicator. Since the user faces the vicinity of the center of the positioning indicator 851 during the calibration process described later, the positioning indicator 851 preferably has a shape captured at the center of the field of view. Figure 20A In this design, the positioning indicator 851 is represented by a circle with a cross mark and a small black circle at the center of the cross mark. However, the shape of the positioning indicator 851 is not limited to this shape. In addition, the positioning indicator can be a rectangular, triangular, star-shaped graphic or a character icon.

[0386] The positioning indicator 851 is captured by the imaging unit 40 of the camera body 1. The display device controller (position calculation unit and distance calculation unit) 801 calculates the distance between the imaging / detection unit 10 and the calibrator 850 based on the captured image and calculates the position coordinates of the positioning indicator 851 appearing in the image area. In this embodiment, the calibrator 850, equipped with the functions of the display device 800, performs these calculations. If the calibrator 850 is not combined with the functions of the display device 800, these calculations are performed by the overall control CPU 101 of the camera body 1.

[0387] Angular velocity sensor 807 can measure the movement of calibrator 850. Based on the measurement value of angular velocity sensor 807, display device controller 801 calculates the movement information related to calibrator 850, as described later.

[0388] In the calibration process described later, the user presses the calibration button 854 while facing the center of the positioning indicator 851. Although in Figure 20A The calibration button 854 is a touch button displayed on the touch-sensitive display unit 803, but button A 802 can be used as a calibration button.

[0389] Next, we will use Figure 21The flowchart details the calibration process performed when extracting images from the ultra-wide-angle image captured by the camera unit 40 based on the user's facial orientation and when applying image processing to the extracted images.

[0390] Figure 21 This is a flowchart illustrating the calibration process performed by the camera body (first calibration unit) 1 and the calibrator 805 according to this embodiment.

[0391] To aid in the explanation, the steps for the camera body 1 or calibrator 850 to receive instructions from the user are included in the box where the operating subject is the user. Furthermore, in Figure 21 In this context, the steps performed by the display device controller 801 of the calibrator 850 in response to user instructions are included in the section where the calibrator 850 is the operating entity. Similarly, in Figure 21 In the context of the camera body 1, the steps performed by the overall control CPU 101 in response to user instructions are included in the box where the operating subject is the camera body 1.

[0392] Specifically, Figure 21 The camera body 1 is the main operator in steps S3104 and S3108. The user is the main operator in steps S3101, S3105, and S3106. Furthermore, the calibrator 850 is the main operator in steps S3102, S3103, S3106a, S3107, S3107b, and S3110.

[0393] In this process, if the calibrator 850 is not powered on, in step S3101, the user powers on the calibrator 850 by operating button A 802. Similarly, if the camera body 1 is not powered on, the user powers on the camera body 1 by switching the power switch 11 to the ON position. Afterwards, the user establishes a connection between the calibrator 850 and the camera body 1. Upon establishing this connection, the display device controller 801 and the overall control CPU 101 enter calibration mode respectively.

[0394] Furthermore, in step S3101, the user wears the camera body 1 and adjusts the length of the straps 82L and 82R and the angle of the camera body 1 so that the camera body 1 is positioned in a suitable location and the camera / detection unit 10 can capture images.

[0395] In step S3102, the display device controller (first display unit) 801 displays the positioning indicator 851 on the display unit 803.

[0396] In the next step S3103, the display device controller 801 specifies that the user should maintain the calibrator 850 in a designated position as indicated by the display 855. Figure 22AIn this embodiment, five positions are sequentially designated as the front, upper right, lower right, upper left, and lower left. These designated positions can be set to other positions, as long as calibration is available.

[0397] In step S3104, the overall control CPU 101 activates the camera unit 40 to enable camera operation and activates the face direction detection unit 20 to enable detection of the user's face direction.

[0398] In step S3105, the user holds the calibrator 850 at the specified position as specified in step S3103.

[0399] In the next step S3106, while maintaining the position of the calibrator 850 in the designated position, the user faces the direction of the positioning index 851 so that the user's field of view center is aligned with the positioning index 851, and presses the calibration button 854.

[0400] In step S3106a, the display device controller (second display unit) 801 determines whether the user is looking at the center 852 of the positioning indicator 851, that is, whether the user's field of view center is consistent with the center 852 of the positioning indicator. If it is determined that the user is looking at the center 852 of the positioning indicator ("yes" in step S3106a), the display device controller 801 notifies the user of the start of calibration for the specified position via the indicator display 855 in step S3107, and redisplays the calibration button 854. If the determination result in step S3106a is "no", the user repeats the process from step S3105 onwards.

[0401] If the user presses the calibration button 854 in step S3107a, the display device controller 801 sends a calibration instruction to the camera body 1 in step S3107b.

[0402] In step S3108, the overall control CPU (acquisition / detection unit) 101, in response to a calibration instruction from the calibrator 850, acquires an ultra-wide-angle image captured by the camera unit 40, including the positioning indicator 851, and detects the facial orientation using the facial orientation detection unit 20. Then, the overall control CPU (generation unit) 101 calculates positional coordinate information related to the center 852 of the positioning indicator in the acquired ultra-wide-angle image, and generates information showing the relationship between the calculated positional coordinate information and the detected facial orientation.

[0403] The following will use Figures 22A to 22F To explain the details of the processing in steps S3103 to S3108. Figures 22A to 22FThis diagram illustrates the calibration operation performed in the user's frontal orientation. The calibration operation is performed so that the center position of the target field of view 125 in the image captured by the imaging unit 40 of the camera body 1 is consistent with the center position of the user's field of view in a natural state.

[0404] Figure 22A This is shown during the calibration operation in the user's frontal orientation. Figure 21 A diagram of the screen displayed on the display unit 803 of the calibrator 850 in step S3103.

[0405] like Figure 22A As shown, the calibrator 850 displays a positioning indicator 851 and an indication display 855 indicating the position where the user should position the positioning indicator 851 on the display unit 803.

[0406] Instruction display 855 is a string instructing the user to center the positioning indicator 851 in the user's field of vision when facing forward. It should be noted that the instruction displayed as instruction display 855 is not limited to a string. For example, the instruction can be displayed using diagrams, pictures, or moving images. Furthermore, instruction display 855, such as in a so-called general tutorial, can be displayed first, followed by the positioning indicator 851.

[0407] Figure 22B It shows the user based on as Figure 22A The diagram shows the indicator displayed by 855, indicating that the calibrator 850 is in the positive position.

[0408] In step S3105, the user, based on... Figure 22A The indicator 855 shows the instructions displayed while keeping the calibrator 850 facing forward. Then, in step S3106, the user holds the calibrator 850 so that the positioning index 851 is aligned with the center of the user's field of vision when facing forward, and the user presses the calibration button 854. Figure 22A In response to the pressing of calibration button 854, the judgment in step S3106a is performed. The specific process of this judgment method will be explained later. If the judgment result in step S3106a is "yes", the display device controller 801 will... Figure 22A The indicated display 855 changes to a notification "Start calibration for front orientation" and displays the calibration button 854.

[0409] Then, the user confirms Figure 22AThe indicated display 855 changes to a notification "Start calibration for frontal orientation" followed by pressing the calibration button 854 (step S3107a). In response to the pressing of the calibration button 854, a calibration instruction is sent to the camera body 1 in step S3107b. And in step S3108, the imaging unit 40 acquires the captured image.

[0410] Figure 22C It is shown in Figure 22B A schematic diagram of the entire ultra-wide-angle image captured by camera lens 16 in the specified state. Figure 22D This shows the correction Figure 22C A schematic diagram of the image obtained by aberrations of the ultra-wide-angle image shown.

[0411] In addition, in response to Figure 22B In the state where the user presses the calibration button 854, the facial orientation detection unit 20 obtains the facial orientation in step S3108.

[0412] Figure 22E This is shown during the calibration operation in the user's frontal orientation. Figure 21 A schematic diagram of the facial orientation image recorded by the facial orientation detection unit 20 in step S3108.

[0413] If in use Figures 8G to 8K In the first embodiment, the facial orientation detection unit 20 uses the distances and angles of the chin positions 207, 207r, and 207u relative to the throat position 206 to calculate the angles in the horizontal and vertical directions of the face. However, as with the image center, the distances and angles of the chin positions 207, 207r, and 207u relative to the throat position 206 also have individual and adjustment differences due to factors such as the user's physical condition, and therefore these distances and angles are not fixed. Therefore, in this embodiment, the relationship between the chin position and the throat position 206 when the calibration button 854 is pressed is defined as the value when the user places the center of the field of view in front of them (reference position). This can be used as information (reference position information) to correctly calculate the user's facial orientation regardless of individual and adjustment differences.

[0414] Return to Figure 21 In step S3109, the overall control CPU 101 determines whether it is ready to calibrate the frontal orientation. That is, it determines whether it has obtained the information required to calculate the chin position 207, the throat position 206, and the positioning index center 852.

[0415] At this point, if the required information has not been obtained, it is determined that calibration is not ready ("No" in step S3109), and the operations from step S3102 onwards are repeated to obtain the missing information from the required information. If the required information has not been obtained, not all operations from step S3102 are necessary. The operations used to obtain the missing information can be performed again.

[0416] The determination in step S3106a is performed using either the face sensor 806 installed in the calibrator 850 or the built-in camera 805. The specific process of this determination method will be explained below using the case where a calibration operation for the frontal orientation is performed using the built-in camera 805. Although the case using the face sensor 806 differs from the case using the built-in camera 805 in terms of the dimension of information (two-dimensional or three-dimensional), the basic approach is the same. Therefore, a detailed description of the case using the face sensor 806 is omitted. When the face sensor 806 is used in the determination in step S3106a, the face orientation detection unit 20 of the camera body 1 does not perform face detection of the user being illuminated by infrared light 823 while the user is being irradiated by infrared light 823 from the face sensor 806. This is to prevent interference between infrared light 23 and 823.

[0417] First, in step S3106, the user presses... Figure 22A In the case of calibration button 854, display device controller 801 obtains a built-in camera image 858 of the user's appearance by taking an image with the built-in camera (face detection unit) 805. Figure 22F In addition, the display device controller 801 detects position information related to the front of the neck 201, the chin 203, the face 204 including the nose, and the camera / detection unit 10 (camera unit 40) from the acquired built-in camera image 858.

[0418] In step S3106a, the display device controller (judgment unit) 801 uses the position information detected from the built-in camera image 858 to determine whether the user is looking at the positioning indicator center 852 of the positioning indicator 851 at the center of the field of view.

[0419] As a result of this determination, if it is determined that the user is looking in a different direction, the display device controller 801 displays a message indicating that correct information cannot be obtained as an indication display 855. This can instruct the user to perform the calibration operation again.

[0420] If the camera / detection unit 10 is tilted beyond a certain angle, or if the face detection window 13 is obstructed or dirty, the display device controller 801 can determine, using the built-in camera image 858, that a correct calibration operation cannot be performed. In this case, the display device controller 801 can display a message indicating that correct information cannot be obtained as an indication display 855.

[0421] Furthermore, the built-in camera image 858 obtained in step S3106a and the ultra-wide-angle image obtained in step S3108 can be used to obtain the information required for parallax correction mentioned later in the fifth embodiment.

[0422] Specifically, before the camera unit 40 captures the positioning indicator 851 in step S3108, information related to the size of the positioning indicator 851 (horizontal width L851a and vertical width L851b) is sent from the calibrator 850 to the camera body 1. Therefore, the overall control CPU 101 can calculate the distance between the camera / detection unit 10 and the positioning indicator 851 using the information related to the size of the positioning indicator 851 and the image of the positioning indicator 851 appearing in the ultra-wide-angle image obtained in step S3108. Since the positioning indicator 851 and the built-in camera 805 are installed in the same housing as the calibrator 850, and... Figure 22B With the calibrator 850 facing the user, the distance between the built-in camera 805 and the camera / detection unit 10 is equal to the distance between the positioning index 851 and the camera / detection unit 10.

[0423] Similarly, in step S3106a, the built-in camera 805 is used to take pictures. Figure 22F Before displaying the image from the built-in camera, information related to the size of the camera / detection unit 10 is sent from the camera body 1 to the calibrator 850. Thus, the display device controller (vertical distance calculation unit) 801 can use the information related to the size of the camera / detection unit 10 and the... Figure 22F The image from the built-in camera image 858 shown in the image of the camera / detection unit 10 is used to estimate the vertical distance 5070 between the optical axis center of the camera lens 16 and the user's viewing position. Additionally, the display device controller 801 can estimate the distance 2071 between the camera lens 16 and the user's chin 203. Distance 2071 can be the distance between the face orientation detection window 13 and the chin 203.

[0424] In order for the facial orientation detection unit 20 to calculate the user's throat position 206 and chin position, the user's face needs to be separated from the facial orientation detection window 13 by a distance greater than a certain distance, according to the design of the facial orientation detection unit 20. Therefore, this estimation result can be used as one of the judgment conditions for determining whether the facial orientation detection unit 20 can correctly detect the facial orientation.

[0425] Return to Figure 21 When the overall control CPU 101 determines that it has obtained the required information and completed the preparation for calibration in the front direction, it causes the process to proceed to step S3110.

[0426] In step S3110, the display device controller (first calibration unit) 801 calculates the information required to shift the extraction center position to absorb individual differences and adjust for differences, and shifts the extraction center position based on the information.

[0427] The details of the calculation in step S3110 will be explained below. If the user is in an ideal state according to the design values ​​and is ideally wearing the camera body 1, then in Figure 22C The center 856 of the ultra-wide-angle image obtained in step S3108 should be almost identical to the center 852 of the positioning indicator appearing in the ultra-wide-angle image. However, due to individual differences and adjustment differences caused by factors such as the user's physical condition, the center 856 of the ultra-wide-angle image is usually not identical to the center 852 of the positioning indicator.

[0428] For the user, it is preferable to make the extraction center position coincide with the center of the user's field of view in the current posture or operation (i.e., the center of the positioning index 852 in the ultra-wide-angle image), rather than with the center 856 of the ultra-wide-angle image shown by the camera body 1.

[0429] Therefore, the deviation of the positioning index center 852 from the center 856 of the ultra-wide-angle image is measured, and the extracted center position is shifted to a position based on the positioning index center 852, which is different from the center 856 of the ultra-wide-angle image. Furthermore, the facial orientation detected by the facial orientation detection unit 20 is also shifted in a similar manner at this time.

[0430] Will be through reference Figure 22C and Figure 22D To illustrate the specific offset method, the deviation of the positioning index center 852 from the center 856 of the ultra-wide-angle image is measured, and as follows... Figure 22C As shown, the measured deviation is divided into lateral deviation 857a and vertical deviation 857b. After appropriate transformation based on the projection method of the entire field angle, the offset is determined based on the deviations 857a and 857b.

[0431] In addition, such as Figure 22D As shown, the offset can be determined after applying appropriate transformation processing to the ultra-wide-angle image according to the projection method. That is, the deviation of the center 856a in the transformed captured image from the center 852 of the positioning index is measured. This deviation is then divided into a lateral deviation 857c and a vertical deviation 857d. The offset can then be determined based on the deviations 857c and 857d.

[0432] The processing load and purpose of the camera system can be taken into account. Figure 22C and Figure 22D Choose any offset method from the methods shown.

[0433] By performing the above-described calibration operation on the frontal orientation, regardless of individual differences and adjustment differences, the facial orientation of the user wearing the camera body 1, the field of view center in the facial orientation within the ultra-wide-angle image, and the facial orientation detected by the facial orientation detection unit 20 are appropriately correlated.

[0434] Of the five directions (front, top right, bottom right, top left, and bottom left), the calibration procedure for the front direction has been described so far. Similar calibration procedures need to be performed for the remaining four directions.

[0435] Therefore, in Figure 21 If step S3110 is completed, the process proceeds to step S3111. In step S3111, if there is a direction among the five directions that has not undergone calibration, the target direction of the calibration operation is changed, and the process returns to step S3103. Thus, the calibration operation is similarly repeated for the other four directions, excluding the completed frontal direction.

[0436] Despite Figure 21 The process is not shown in the diagram, but if it is determined in step S3111 that there is no direction for which no calibration operation has been performed, the process ends as is.

[0437] Figures 23A to 23E This diagram illustrates the calibration operation for the user's upper right direction (upper right direction in an ultra-wide-angle image). Figures 23A to 23E respectively with Figures 22A to 22E Correspondingly, the basic operations are also the same. Therefore, common explanations have been omitted.

[0438] like Figure 23A As shown, indicator display 855 displays the following string, which instructs the user to position the positioning indicator 851 at the center of the user's field of view when the user's face is turned to the upper right.

[0439] Figure 23B It shows the user based on the Figure 23AThe diagram shows the indications in Figure 855, which indicates that the calibrator 850 is held in the upper right position. Figure 23C It is shown in Figure 23B A schematic diagram of the entire ultra-wide-angle image captured by camera lens 16 in the specified state.

[0440] like Figure 23C As shown, firstly, the deviation between the center 856 of the ultra-wide-angle image and the center 852 of the positioning index is measured according to a specific offset method. Then, the measured deviation is divided into radial deviation 857e and angular deviation 857f. After appropriate transformation processing based on the projection method of the entire field angle, the offset is determined based on the deviations 857e and 857f.

[0441] In addition, such as Figure 23D As shown, the offset can be determined after applying appropriate transformation processing to the ultra-wide-angle image according to the projection method. That is, the deviation of the center 856a in the transformed captured image relative to the positioning index center 852 is measured. The deviation is then divided into a radial deviation 857g and an angular deviation 857h. The offset can then be determined based on the deviations 857g and 857h.

[0442] use Figures 22A to 22E The offset is determined by dividing the deviation into lateral and vertical deviations. In contrast, using... Figures 23A to 23E The offset is determined using a method that divides the deviation into radial and angular deviations. The method is varied only for ease of explanation, and any method can be used.

[0443] In addition, such as Figure 23E As shown, the facial orientation detection unit 20 obtains the throat position 206 and chin position 207ru required to calculate the facial orientation when the face is facing upward to the right. Therefore, regardless of individual differences and adjustment differences among users, the facial orientation of the user when looking in a direction (upward to the right in this case) toward the center of the positioning index 852 can be accurately measured.

[0444] As mentioned above, in Figure 21 In the calibration process shown, calibration operations are performed in the upper right, lower right, upper left, and lower left directions, excluding the frontal direction. Therefore, when the user turns their head in any of the up, down, right, or left directions, the facial orientation detection unit 20 can accurately detect the direction of the user's turn. Thus, regardless of individual differences or adjustment variations, the user can appropriately use the camera body 1.

[0445] The above description simplifies the process by explaining how to repeat the calibration operation in five directions (front, top right, bottom right, top left, and bottom left).

[0446] However, the calibration operation is not limited to this method. For example, the following method can be used. That is, the user continuously moves the calibrator 850 along a zigzag, spiral, or polygonal trajectory according to the instructions displayed 855. At the same time, the user continuously captures the positioning index 851 displayed on the calibrator 850 at the center of the field of view. In this method, the display device controller 801 sends calibration instructions multiple times to the camera body 1 while the calibrator 850 is moving.

[0447] Whenever a calibration instruction is received, the overall control CPU 101 obtains the facial orientation detected by the facial orientation detection unit 20 and the positional coordinates related to the center 852 of the positioning indicator in the ultra-wide-angle image captured by the camera unit 40, and saves both as historical information. Then, the overall control CPU 101 calculates the relationship between the extracted center position of the image and the user's facial orientation by combining information extracted from the acquired historical information. Furthermore, in this method, the information extracted from the historical information can be limited to information obtained when the user is looking at the positioning indicator 851. This information is limited using information obtained by the calibrator 850 during the movement of the calibrator 850 related to the built-in camera 805 and the facial sensor 806. Thus, information obtained when the user is looking elsewhere is no longer extracted from the historical information, which improves the accuracy of the relationship calculation.

[0448] Furthermore, the display device controller 801 can send the measurement value of the angular velocity sensor 807 along with a calibration instruction to the camera body 1. In this case, the overall control CPU 101 obtains movement information from the sent measurement value of the angular velocity sensor 807, showing the movement trajectory of the calibrator 850 using the user's calibrator, as well as the position and posture of the calibrator 850. This movement information is also saved as historical information. Thus, based on the movement information based on the measurement value of the angular velocity sensor 807, the facial direction detected by the facial direction detection unit 20, and the position coordinate information related to the center of the positioning index 852 in the ultra-wide-angle image captured by the camera unit 40, the calibration operation can be performed easily and accurately.

[0449] In this case, the movement information based on the measurements from the angular velocity sensor 807 should be consistent with the movement information based on the position coordinates related to the positioning index 851. Therefore, when using the measurements from the angular velocity sensor 807, it is necessary to synchronize the communication between the camera body 1 and the calibrator 850.

[0450] As described above, the second embodiment describes a calibration method that can correlate the user's facial orientation with the center position of the target field of view 125 set in the ultra-wide-angle image, regardless of individual differences and adjustment variations. On the other hand, the present invention is not limited to the various configurations illustrated in the second embodiment, and various modifications can be utilized within the scope of the invention.

[0451] Next, the third embodiment will be described. In the third embodiment, using Figures 24A to 29 This section details the method for detecting the deviation of the camera body 1 relative to the user and the operation during deviation detection.

[0452] This embodiment will be described essentially as a derivative of the first embodiment. Therefore, structures in the camera system of the third embodiment that are identical in structure to those in the camera system of the first embodiment are indicated by the same reference numerals, and repeated descriptions are omitted. Different structures will be described with added details.

[0453] The camera device in this embodiment detects the user's facial orientation by observing the user's head from the user's throat, and obtains an output image by extracting an image from an ultra-wide-angle image based on the detection result. Therefore, when the user does not wear the camera body 1 equipped with the facial orientation detection unit 20 and the camera unit 40 correctly, or when there is a deviation relative to the position determined by calibration, the facial orientation cannot be detected correctly and the output video image will be tilted. Therefore, it is preferable to detect the positional deviation of the camera body 1 relative to the user and correct or warn accordingly.

[0454] Since the camera device in this embodiment is assumed to be a halter-neck type of clothing, just like in the first embodiment, there is a high possibility of lateral and pitch deviations around the main axis of the user's line of sight when looking at the front.

[0455] Figure 24A and Figure 24B This is a schematic front view showing a user wearing the camera body 1. Figure 24A This shows the state in which the user is correctly wearing the camera body 1. Figure 24B This shows the state in which the camera body 1 tilts relative to the user.

[0456] Figure 25A and Figure 25B This is a schematic side view showing a user wearing the camera body 1. Figure 25A This shows the state in which the user is correctly wearing the camera body 1. Figure 25B This shows the state in which the camera body 1 is tilted relative to the user.

[0457] Figure 26 This is a functional block diagram illustrating the camera body 1 according to this embodiment. In the following text, it will be used... Figure 26This is a brief explanation of the processing performed by the camera body 1. Details will be explained later.

[0458] like Figure 26 As shown, the camera body 1 includes a face orientation detection unit 20, a recording direction / field angle determination unit 30, an imaging unit 40, an image extraction / display unit 50, a main recording unit 60, and a transmission unit 70. Furthermore, the camera body 1 includes a second controller 111, an offset detection unit 9021, a face orientation correction unit (observation direction correction unit) 9022, and a recording angle determination unit (recording image correction unit) 9031. These functional blocks are controlled by the overall control CPU 101 that controls the entire camera body 1.

[0459] The face orientation detection unit 20 detects the state of the face and transmits the state of the face to the deviation detection unit 9021. In addition, the face orientation detection unit 20 estimates the observation direction based on the detected state of the face and transmits the observation direction to the recording direction / field angle determination unit 30.

[0460] The deviation detection unit 9021 detects the tilt and yaw deviations of the camera body 1 relative to the user based on the facial state detected by the facial orientation detection unit 20. It then transmits the tilt deviation to the recording angle determination unit 9031 and the tilt and yaw deviations to the facial orientation correction unit 9022. Furthermore, when a tilt or yaw deviation is detected, the deviation detection unit 9021 notifies the second controller 111 of the deviation information.

[0461] When deviation information is received, the second controller 111 uses the speaker 105 and vibrator 106 of the camera body 1 to notify the user of a warning. Alternatively, when deviation information is received, the second controller 111 notifies the display device 800 of the deviation information via the transmitting unit 70 to notify the user of a warning using the display unit 803, speaker 815, vibrator 816, or LED 817.

[0462] At this time, one or more of the following components are used as warning elements: the speaker 105 of the camera body 105, the vibrator 106, the display unit 803 of the display device 800, the speaker 815, the vibrator 816, and the LED 817.

[0463] The recording angle determination unit 9031 calculates the tilt correction angle of the recorded image based on the tilt deviation and transmits the tilt correction angle to the image extraction / display unit 50.

[0464] The face orientation correction unit 9022 calculates the correction amount for the observation direction based on the tilt deviation and pitch deviation, and transmits the correction amount for the observation direction to the recording orientation / field angle determination unit 30.

[0465] The recording orientation / field angle determination unit 30 performs various calculations based on the observation orientation estimated by the face orientation detection unit 20 and the correction amount of the observation orientation calculated by the face orientation correction unit 9022, and determines information related to the position and region of the image to be extracted from the ultra-wide-angle image from the camera unit 40. Then, the recording orientation / field angle determination unit 30 transmits the determined information to the image extraction / display unit 50.

[0466] The camera unit 40 converts light from the subject into an image and transmits the image to the image extraction / display unit 50. The image extraction / display unit 50, using information from the recording direction / field angle determination unit 30 and the recording angle determination unit 9031, extracts only the image in the direction the user is looking from the image output from the camera unit 40, displays the extracted image, and transmits the displayed image to the main recording unit 60. Since the subsequent processing is the same as in the first embodiment, descriptions of these processes are omitted.

[0467] Figures 27A to 27C This is a diagram showing an image of the user as seen from the face-oriented detection window 13. Figure 27A This shows the state in which the user is correctly wearing the camera body 1. Figure 27B This shows the state in which the camera body 1 tilts relative to the user. Figure 27C This shows the state in which the camera body 1 is tilted relative to the user.

[0468] like Figure 27A As shown, when the user wears it correctly, the throat position 9206a (the center of head rotation) is located in the center of the image. Figure 27B As shown, when a tilt deviation occurs, the throat position 9206b is located on the right side of the image. Therefore, when the deviation of the throat position 9206b relative to the left or right center of the image exceeds a predetermined threshold, it can be determined that the camera body 1 has a tilt deviation relative to the user. In other cases, when the deviation of the throat position 9206b relative to the position determined through calibration exceeds a predetermined threshold, it can be determined that the camera body 1 has a tilt deviation relative to the user.

[0469] At this point, since the performance of the infrared focusing lens 26 and the infrared detection device 27 is known, the roll deviation can be calculated based on the deviation of the throat position 9206b.

[0470] Figures 27A to 27C The white box 9220 in the image shows the area where the chin position, known during calibration, might be located. (As shown...) Figure 27A As shown, when the user wears it correctly, the chin position 9207a falls within the white frame 9220. (As indicated...) Figure 27C As shown, when a pan deviation occurs, the chin position 9207c does not fall within the white frame 9220. Therefore, when the chin position 9207c is outside the white frame 9220 where the chin position is known to be during calibration, it can be determined that the camera body 1 has a pan deviation relative to the user.

[0471] At this point, since the performance of the infrared focusing lens 26 and the infrared detection device 27 is known, the pitch deviation can be roughly calculated based on the separation of the throat position 9206c from the white frame 9220.

[0472] Furthermore, despite Figure 27A and Figure 27C Users in the same direction face the same direction, but because Figure 27C The throat position 9206c in the middle moves due to the effect of pitch deviation, thus changing the movement angle θr. Calculate and correct for this change.

[0473] Figures 28A to 28C These are diagrams showing the effective projection area 9122 of the ultra-wide-angle image captured by the camera unit 40 and the target field of view 9125 to be extracted by the image extraction / display unit 50. Figure 28A This shows the state in which the user is correctly wearing the camera body 1. Figure 28B This shows the state in which the camera body 1 tilts relative to the user. Figure 28C This shows the state in which the camera body 1 tilts relative to the user and the target field of view 9125 is corrected using the recording angle determination unit 9031.

[0474] like Figure 28B As shown, when the camera body 1 tilts, it will record an image that is tilted relative to the angle the user is looking at. However, as... Figure 28C As shown, when performing tilt correction on the target field of view 9125, an image close to the image being viewed by the user can be recorded.

[0475] Figure 29 This is a flowchart illustrating the deviation detection process. Using... Figure 29 This will explain the process of deviation detection and processing.

[0476] First, in step S9101, whether the user is exercising is detected based on the output of the angular velocity sensor 107 or the accelerometer 108 (posture detection component). When exercising is detected, deviation detection is stopped in step S9102. Since the position of the camera body 1 relative to the user is unstable during exercise, the effect of deviation detection is unpredictable, and it is troublesome to output a deviation every time the user exercises. To avoid this trouble, deviation detection is stopped during exercise.

[0477] When it is detected in step S9101 that the user is not exercising, in step S9103, the deviation detection unit 9021, as referenced... Figures 27A to 27C The method is used to detect whether roll deviation has occurred. Next, in steps S9104 and S9105, pitch deviation is also detected.

[0478] When a tilt deviation is detected but no pitch deviation occurs, in step S9106, the recording angle determination unit 9031 calculates the tilt correction angle for the extracted image, and in step S9107, the image extraction / development unit 50 performs tilt correction on the extracted image (recorded image tilt correction). This correction enables a reduction in the amount of rotational deviation between the image actually seen by the user and the image to be recorded.

[0479] Furthermore, when a pitch deviation is detected, the facial orientation correction unit 9022 calculates the tilt correction amount and pitch correction amount for the facial orientation in step S9108, and corrects the recording orientation in the recording orientation / field angle determination unit 30 in step S9109. This correction enables the reduction of the deviation between the direction the user is actually facing and the direction to be recorded.

[0480] Furthermore, when either roll deviation or pitch deviation is detected, the target field of view 9125 extracted by the image extraction / development unit 50 can be expanded. Therefore, even if deviation occurs within the camera body 1, as many images as possible can be recorded in the direction the user wants to record.

[0481] When at least one of tilt deviation and pitch deviation is detected, correction for the detected deviation is performed in step S9107 or step S9109. Then, in step S9110, a deviation warning is issued to the user using a warning component. This notification allows the user to notice the deviation of the camera body 1 and supports correction of the deviation. It should be noted that the timing of issuing the deviation warning to the user is not limited to this example. For example, a deviation warning may be issued when tilt deviation and pitch deviation are continuously detected for more than a predetermined time period. A deviation warning may be issued when tilt deviation and pitch deviation are detected more than a predetermined number of times. When no tilt deviation or pitch deviation is detected, normal camera operation continues in step S9111.

[0482] Despite Figure 29 In the treatment of roll deviation, it is possible to detect roll deviation in advance, but it is also possible to detect pitch deviation in advance. The order of detection is irrelevant.

[0483] Although this embodiment describes a method for detecting facial orientation and deviation using an infrared camera, similar to the first embodiment, facial orientation and deviation can also be detected using 3D sensors, such as ToF sensors, or millimeter-wave radar.

[0484] Although this embodiment describes all modes for image tilt correction, face orientation detection correction, and deviation warning, only some of them can be performed. In this case, along from... Figure 29 The process described omits steps that are not executed.

[0485] Next, the fourth embodiment will be described. In the fourth embodiment, a method will be used. Figures 30A to 34 This section details the operation used to correct the output image to the angle at which the user is looking when the user tilts their head horizontally (causing the head to tilt).

[0486] This embodiment will be described essentially as a derivative of the first embodiment. Therefore, structures in the camera system of the fourth embodiment that are identical in structure to those in the camera system of the first embodiment are indicated by the same reference numerals, and repeated descriptions are omitted. Different structures will be described with added details.

[0487] The camera device in this embodiment detects the user's facial orientation by observing the user's head from the user's throat, and obtains an output image by extracting an image from an ultra-wide-angle image based on the detection result. Therefore, the user can record an image very close to the image they are currently viewing. In this type of camera, even when the user tilts their head laterally (causing head tilt), it is desirable to ensure that the angle (tilt angle) of the image the user is looking at matches the angle of the output image.

[0488] Figure 30A This is a diagram showing the user without tilting their head. Figure 30B This is a diagram showing the state of a user tilting their head horizontally (causing the head to tilt).

[0489] Figure 33A and Figure 33B This is a diagram showing the effective projection area 9122 of the ultra-wide-angle image captured by the camera unit 40, and the target fields of view 9125d and 9125e to be extracted by the image extraction / display unit 50. Figure 33A The target field of view 9125d shows the situation when the user is looking straight ahead without tilting their head. Figure 33B The target field of view 9125e shows the situation when the user tilts their head and looks straight ahead.

[0490] like Figure 33A and Figure 33BAs shown, the positional relationship of the effective projection area 9122 relative to the body remains unchanged regardless of the head tilt. However, since the image the user is currently looking at is tilted when the user tilts their head, a tilt rotation needs to be applied only to the target field of view 9125.

[0491] Figure 31 This is a functional block diagram illustrating the camera body 1 according to this embodiment. In the following text, it will be used... Figure 31 This is a brief explanation of the processing performed by the camera body 1. Details will follow.

[0492] like Figure 31 As shown, the camera body 1 includes a face orientation detection unit (distance measurement unit) 20, a recording direction / field angle determination unit 30, an imaging unit 40, an image extraction / display unit 50, a main recording unit 60, a transmission unit 70, a second controller 111, and a recording angle determination unit (tilt angle detection unit) 9031. These functional blocks are controlled by the overall control CPU 101 that controls the entire camera body 1.

[0493] The face orientation detection unit 20 detects the state of the face and transmits the state of the face to the recording angle determination unit 9031. In addition, the face orientation detection unit 20 estimates the observation direction based on the detected state of the face and transmits the observation direction to the recording direction / field angle determination unit 30.

[0494] The recording angle determination unit 9031 calculates the head tilt amount (lateral tilt angle correction amount) based on the facial state received from the facial direction detection unit 20, and transmits the head tilt amount to the image extraction / display unit 50.

[0495] The recording direction / field angle determination unit 30 determines information related to the location and region extracted from the image captured by the imaging unit 40 by performing various calculations based on the observation direction estimated by the face direction detection unit 20. This information is then passed to the image extraction / display unit 50.

[0496] The camera unit 40 converts light from the subject into an image and transmits the image to the image extraction / display unit 50. The image extraction / display unit 50 extracts only the image in the direction the user is looking from the image from the camera unit 40 using information from the recording direction / field angle determination unit 30 and information from the recording angle determination unit 9031, displays the extracted image, and transmits the displayed image to the main recording unit 60. Since the subsequent processing is the same as in the first embodiment, descriptions of these processes are omitted.

[0497] Figure 32A and Figure 32BThis is an image of the user as seen from the face-oriented detection window 13, and it shows the user in relation to... Figure 30B The corresponding angle at which the head is tilted. This will be used... Figure 32A and Figure 32B This explains how to detect head tilt (facial tilt angle).

[0498] Figure 32A The first point 9230 and the second point 9231 (multiple points) are arranged such that the straight line connecting the first point 9230 and the second point 9231 intersects perpendicularly with the straight line connecting the throat position 9206 and the chin position 9207, and is bisected equally by the straight line connecting the throat position 9206 and the chin position 9207. A rough head tilt can be detected by comparing the distance between the first point 9230 and the camera body 1 with the distance between the second point 9231 and the camera body 1.

[0499] on the other hand, Figure 32B The first region 9232 and the second region 9233 (multiple regions) are arranged in a linearly symmetrical manner on both sides of the straight line connecting the throat position 9206 and the chin position 9207. The approximate head tilt can be detected by comparing the average distance between the first region 9232 and the camera body 1 with the average distance between the second region 9233 and the camera body 1.

[0500] By comparison Figure 32A The methods and comparisons of the first and second points in the text Figure 32B The facial tilt angle can be calculated using either the first or second region method.

[0501] Figure 34 This is a flowchart illustrating the tilt correction process from facial tilt angle detection to the extracted image. It will use... Figure 34 This will explain the process of facial tilt correction.

[0502] The camera device in this embodiment is equipped with a first mode that performs tilt correction on the output image and a second mode that does not perform tilt correction. Figure 34 In the processing, step S9201 determines whether the camera device is in the first mode. When the camera device is in the first mode, in step S9202, the recording angle determination unit 9031 uses... Figure 32A and Figure 32B The method shown calculates the facial tilt angle based on information obtained from the facial orientation detection unit 20. Furthermore, when the camera device is in the second mode, the process proceeds to step S9203 without calculating the facial tilt angle.

[0503] In the first mode, in step S9204, it is determined whether the deviation of the facial tilt angle calculated by the recording angle determination unit 9031 from the vertical direction detected by the angular velocity sensor 107 or the acceleration sensor 108 is equal to or greater than a predetermined threshold. Specifically, it is determined whether the tilt angle of the head is relative to a plane along the vertical direction detected by the angular velocity sensor 107 or the acceleration sensor 108 (see...). Figure 30B ).

[0504] When it is determined that the deviation is equal to or greater than the predetermined threshold, in step S9205, the recording angle determination unit 9031 calculates the recording angle based on the calculation result of the facial tilt angle, and the image extraction / display unit 50 performs tilt correction on the extracted image.

[0505] When the angle deviation is determined to be less than a predetermined threshold, the recording angle determination unit 9031 calculates the recording angle based on the vertical direction detected by the angular velocity sensor 107 or the accelerometer 108. Furthermore, through this calculation, the image extraction / display unit 50 performs tilt correction on the extracted image in step S9206. According to the above process, if a user who is not standing upright wants to view an image with ensured horizontality, an image with ensured horizontality can be output. Then, when the user views an image with an angle tilted relative to the horizontal direction, an image at the angle the user is viewing can be recorded.

[0506] Although this embodiment describes a process for determining whether to select one of the two factors—facial tilt angle and vertical direction—it is possible to make a determination based solely on the facial tilt angle without comparing it to the vertical direction. In this case, steps S9204 and S9206 will be omitted.

[0507] Although this embodiment describes a method for detecting facial orientation and deviation using an infrared camera, similar to the first embodiment, facial orientation and deviation can also be detected using 3D sensors, such as ToF sensors, or millimeter-wave radar.

[0508] Next, the fifth embodiment will be described. In the fifth embodiment, a method will be used. Figures 35A to 35C , Figure 36A and Figure 36B This section details the methods used to detect changes in jaw position and the procedures involved in detecting these changes.

[0509] This embodiment will be described essentially as a derivative of the first embodiment. Therefore, structures in the camera system of the fifth embodiment that are identical in structure to those in the camera system of the first embodiment are indicated by the same reference numerals, and repeated descriptions are omitted. Different structures will be described with added details.

[0510] The camera device in this embodiment detects the user's facial orientation by observing the user's head from the throat, and obtains an output image by extracting an image from an ultra-wide-angle image based on the detection result. Since facial orientation is estimated by observing the state of the jaw, errors will occur in the estimation result when the positional relationship between facial orientation and jaw state changes (when the mouth is open and closed). This means that accurate detection is difficult during conversation and meals.

[0511] Figures 35A to 35C This is a schematic side view showing a user wearing the camera body 1. Figure 35A This shows the user looking straight ahead with their mouth closed. Figure 35B It shows the user looking straight ahead with their mouth open. Figure 35C This shows the user looking down at an angle with their mouth closed.

[0512] In fact, Figure 35A and Figure 35B Show the user looking at the front, and Figure 35C This shows the user looking diagonally downwards. However, when viewed from the camera body 1, in... Figure 35A The user in the middle looks positive, and in Figure 35B and Figure 35C The user appears to be looking diagonally downwards. In other words, when the detection results are available when the mouth is closed, the facial orientation can be correctly detected.

[0513] Figure 36A and Figure 36B This is a graph showing an example of the detection results for facial movement in the vertical direction (pant direction). The horizontal axis represents time, and the vertical axis represents the angle in the vertical direction. Figure 36A This shows the detection results when the user's face is facing down while opening and closing their mouth (for talking, etc.). The raw detection value 9400 fluctuates up and down while moving downwards. At this time, the low-pass value 9402 obtained by applying a general low-pass filter to the raw detection value 9400 is the median of the raw detection value 9400. If the low-pass value 9402 is used for detection, facial movement is detected when the user's mouth is half-open. At this time, the judgment value 9401 is obtained when the user's mouth is correctly closed by connecting the peak values ​​of the fine waveform of the raw detection value 9400.

[0514] Figure 36B The results show the detection of a user opening and closing their mouth while changing the amount of mouth opening, while looking directly at them. (As shown from...) Figure 36BIt is understood that even though the user is looking directly at the face, the output image will jitter vertically when the detection orientation is determined based on the low-pass value 9402. Similarly, in this case, when the face orientation is determined based on the judgment value 9401 obtained by connecting the peak value of the fine waveform of the raw detection value 9400, the output image will remain facing forward. Therefore, when the raw data for detecting the face orientation in the vertical direction (pant angle) vibrates at frequencies within a predetermined range, the judgment value 9401 is obtained by connecting the maximum value (upper peak) of the waveform across each cycle. As a result, the face orientation can be correctly detected even during conversation and meals.

[0515] In this embodiment, it is assumed that the frequency of opening and closing the mouth during conversation and eating is approximately 1 Hz to 10 Hz.

[0516] like Figures 35A to 35C As shown, the camera body 1 is equipped with a user-side microphone 9019 (sound detection component) for detecting the user's vocalizations and chewing sounds. This can be based on... Figure 36A and Figure 36B The frequency of the raw detection value 9400 shown is used to determine conversation and eating. In addition, when the user-side microphone 9019 detects a sound equal to or greater than a predetermined threshold, the facial direction can be determined using the judgment value 9401, which is a fine waveform connected to the raw detection value 9400.

[0517] Furthermore, when performing the calibration as in the second embodiment, an instruction can be provided from the display device 800 or the calibrator 850 to prompt calibration with the mouth closed. This allows for improved accuracy of calibration and pitch angle correction. For example, the calibrator 850 can announce an audio message during calibration. This prompts the user to perform calibration with their mouth closed.

[0518] Next, the sixth embodiment will be described. As described in the second embodiment, due to the user's physique and the mounting state of the camera body 1, individual differences exist in the relationship between the facial orientation detected by the facial orientation detection unit 20 and the center position of the target field of view 125 (the image extraction position). Therefore, calibration processing is required to associate the image extraction position with the facial orientation. In addition to calibration for the frontal orientation, calibration operations are also required for the upper right, lower right, upper left, and lower left orientations. Such operations can be cumbersome for the user. The sixth embodiment reduces the time and effort required for the user in the calibration process.

[0519] This embodiment will be described essentially as a derivative of the second embodiment. Therefore, structures in the camera system of the sixth embodiment that are identical to those in the camera systems of the first and second embodiments are indicated by the same reference numerals, and repeated descriptions are omitted. Different structures will be described with added details.

[0520] Figure 37 This figure illustrates details of the calibrator 850 used in the calibration process according to the sixth embodiment. In this embodiment, the calibration system includes a camera body (video recording device) 1 and a calibrator 850. The display device 800 also serves as the calibrator 850. It should be noted that, as with the first and second embodiments, the calibrator 850 can be a dedicated device, a general-purpose smartphone, or a tablet terminal.

[0521] like Figure 37 As shown, the calibrator 850 includes a button A 802, a display unit 803, a built-in camera 805, a face sensor 806, a positioning indicator 851, a camera image display 4001, and a message display 4002. Although in this embodiment the positioning indicator 851, camera image display 4001, and message display 4002 are to be displayed on the display unit 803, they are not limited thereto. Thus, the display unit 803 serves as both a received image display unit for displaying the camera image display 4001 and an indicator display unit for displaying the positioning indicator 851. This omits the need for a separate indicator display unit from the display unit 803, thereby simplifying the structure of the calibrator 850 and contributing to miniaturization.

[0522] As described in the first embodiment, the ultra-wide-angle image captured by the camera body 1 is transmitted by the transmitting unit 70 and received by the high-speed wireless communication unit 872 or the low-power wireless communication unit 871 of the calibrator 850. Then, an image drawn by superimposing various information onto the ultra-wide-angle image captured by the camera body 1 (image capturing unit) is displayed as a camera image display 4001. Figure 37 As shown, the image includes calibrator 850 as one of various pieces of information. Messages prompting user action and error messages are displayed as message display 4002.

[0523] First, the processing of the camera body 1 will be explained. Figure 38 This is a flowchart illustrating the processing of the camera body 1 in the calibration process according to this embodiment. It should be noted that each process in this flowchart is implemented because the overall control CPU 101 runs the program stored in the internal non-volatile memory 102.

[0524] When the calibration process begins, firstly, in step S4101, the camera unit 40 captures an ultra-wide-angle image.

[0525] Next, in step S4102, the aberrations of the ultra-wide-angle image captured (obtained) in step S4101 are corrected. Figure 40AThis is a schematic diagram illustrating an example of an ultra-wide-angle image captured by camera body 1 and whose aberrations have been corrected during calibration processing. As described in the second embodiment, the user holds the calibrator 850 in front of their body during calibration processing. Therefore, as... Figure 40A As shown, the calibrator 850 and the user's hand 4301 holding the calibrator 850 appear in the image captured by the camera body 1 worn at the collarbone position.

[0526] Next, in step S4103, the transmitting unit 70 of the camera body 1 sends the image corrected in step S4102 to the calibrator 850.

[0527] Next, in step S4104, the transmitting unit (coordinate transmitting unit) 70 sends the target position for calibration to the calibrator 850. The calibrator 850 receives the target position via the high-speed wireless communication unit 872 or the low-power wireless communication unit 871 (coordinate receiving unit). Hereinafter, "target position for calibration" refers to the position the user moves the calibrator 850 towards for calibration; that is, it is the coordinate information of the destination of the calibrator 850's movement. In this embodiment, the center, upper left, lower left, upper right, and lower right of the camera image display 4001 are set (designated) as the target positions. However, the target positions are not limited to this example. Furthermore, target positions can be added or omitted if necessary.

[0528] Next, in step S4105, the overall control CPU (index detection unit) 101 detects the positioning index 851 displayed on the calibrator 850 based on the ultra-wide-angle image corrected in step S4102, and calculates the coordinates of the positioning index 851. For example, it should be noted that the positioning index 851 can be detected by searching for image regions whose characteristic features, such as color and shape, are consistent with the positioning index 851. Alternatively, the positioning index 851 can be detected using machine learning methods such as deep learning.

[0529] Next, in step S4106, the overall control CPU 101 determines whether the positioning indicator 851 detected in step S4105 is located at the target position (predetermined position). This determination is performed by judging whether the coordinates of the positioning indicator 851 calculated in step S4105 are located at or near the coordinates of the target position. Then, as a result of the determination in step S4106, if it is determined that the positioning indicator 851 is located at the target position, the process proceeds to step S4107. Then, the overall control CPU (calibration unit) 101 begins calibration. On the other hand, as a result of the determination in step S4106, if it is determined that the positioning indicator 851 is not located at the target position, the process proceeds to step S4110. When it is determined that the positioning indicator 851 is not located at the target position, the calibrator 850 gives the user an instruction to move the positioning indicator 851 to the predetermined position. This will be explained later by referring to... Figure 39 To illustrate.

[0530] In step S4107, following the determination in step S4106, the facial orientation detection unit 20 detects the facial orientation (the user's facial orientation). This facial orientation detection process is the same as the process in step S200 described in the first embodiment.

[0531] In step S4108, following the determination in step S4106, the overall control CPU 101 calculates correction information. This correction information is the same as that described in the second embodiment, and it shows the relationship between the coordinates of the positioning indicator 851 detected in step S4105 and the facial direction detected in step S4107. It should be noted that in this embodiment, the description related to determining whether the user is looking directly at the positioning indicator 851 at the center of the visual field is omitted for simplicity. However, as in the second embodiment, correction information can be calculated even if the user is looking directly at the positioning indicator 851 at the center of the visual field.

[0532] Next, in step S4109, the overall control CPU 101 updates the target position. In this embodiment, the target position is updated in the order of center, upper left, lower left, upper right, and lower right of the camera image display 4001. However, the target position is not limited to this example.

[0533] Next, in step S4110, the overall control CPU 101 determines whether correction information has been calculated at all target locations. As a result of the determination in step S4110, if it is determined that correction information has been calculated at all target locations, the process proceeds to step S4111. On the other hand, as a result of the determination in step S4110, if it is determined that correction information has not been calculated at all target locations, the process returns to step S4101, and the steps starting from step S4101 are executed sequentially.

[0534] In step S4111, following the determination in step S4110, the correction information obtained up to step S4110 is stored in the internal non-volatile memory 102 or the large-capacity non-volatile memory 51 (correction information storage unit) of the camera body 1. Therefore, for example, the stored correction information can be sent to the calibrator 850 at any time. In this case, the correction information is transmitted via the transmitting unit (correction information transmitting unit) 70 of the camera body 1 and received by the high-speed wireless communication unit 872 or the low-power wireless communication unit 871 (correction information receiving unit) of the calibrator 850. Then, the correction information is also stored in the large-capacity non-volatile memory (correction information storage unit) 814 of the calibrator 850.

[0535] Next, in step S4112, the sending unit 70 notifies the calibrator 850 that the calibration process is complete and the process ends.

[0536] Next, the processing of calibrator 850 will be explained. Figure 39 This is a flowchart illustrating the processing of the calibrator 850 in the calibration process according to this embodiment. It should be noted that the processes in this flowchart are implemented because the display device controller 801 runs the program stored in the internal non-volatile memory 812.

[0537] When calibration begins, firstly, in step S4201, the display device controller 801 displays the positioning indicator 851 on the display unit 803. The processing in step S4105 detects the positioning indicator 851 displayed on the display unit 803 in step S4201.

[0538] Next, in step S4202, the display device controller 801 displays a message on the display unit 803 prompting the user to perform an operation. Figure 41A This is a schematic diagram showing the display state of the display unit 803 when step S4202 is completed. For example... Figure 41A As shown, a positioning indicator 851 arranged near the center of the display unit 803 and a message display 4002 arranged below the positioning indicator 851 are displayed on the display unit 803. The message display 4002 displays a message prompting the user to move the calibrator 850. It should be noted that the position and shape of the positioning indicator 851 are not limited to... Figure 41AThe position and shape shown are acceptable as long as they can be detected by the camera body 1. Furthermore, if the positioning indicator 851 is located on the periphery of the ultra-wide-angle image, the shape of the positioning indicator 851 may be distorted. In this case, the display unit (indicator display unit) 803 changes the shape or color of the positioning indicator 851 over time, making the positioning indicator 851 detectable. For example, the shape of the positioning indicator 851 is changed by repeatedly switching between rectangles and circles. Furthermore, the color of the positioning indicator 851 is changed, for example, by repeatedly switching between colored (e.g., red) and non-colored (e.g., black).

[0539] Next, in step S4203, the high-speed wireless communication unit 872 or the low-power wireless communication unit 871 receives the ultra-wide-angle image from the camera body 1. This image is transmitted in step S4103.

[0540] Next, in step S4204, the high-speed wireless communication unit 872 or the low-power wireless communication unit 871 receives the coordinates of the target position from the camera body 1. These coordinates of the target position are transmitted in step S4104.

[0541] Next, in step S4205, the display device controller (drawing unit) 801 draws a marker for the coordinates of the target position received in step S4204, superimposed on the ultra-wide-angle image received in step S4203. The coordinates (coordinate information) of the target position indicate to the user the destination of the calibrator 850's movement, i.e., the coordinates of the marker used when moving the calibrator 850. Furthermore, although the marker for the coordinates of the target position is, for example, a circle concentrically arranged with the center of the camera image display 4001 and having a predetermined radius, the shape is not limited to a circle. For example, a quadrilateral or other shapes may be used.

[0542] Figure 40B This is a schematic diagram illustrating an image where the target position is drawn as a circular marker 4302 and superimposed on the ultra-wide-angle image, with the target position located in the center of the camera image display 4001. (As shown) Figure 40B As shown, in the camera image display 4001, a mark 4302 is drawn with a dashed line, except for the user's hand 4301. Mark 4302 prompts the user to move the calibrator 850 so that the calibrator 850 is positioned inside the mark 4302. Therefore, the radius of the mark 4302 is determined taking into account the size of the calibrator 850. Then, a message prompting the user to move the calibrator 850 inside the mark 4302 is displayed as message display 4002. The user, having acknowledged this message, can move the calibrator 850 inside the mark 4302 accordingly.

[0543] Next, in step S4206, the display device controller 801 displays the ultra-wide-angle image drawn in step S4205 as a camera image display 4001 at a predetermined position on the display unit (image receiving display unit) 803. Figure 41B This is a schematic diagram showing the display state of the display unit 803 when step S4206 is completed. For example... Figure 41B As shown, a circular marker 4302 is drawn as an image superimposed on an ultra-wide-angle image as a camera image display 4001. In this way, the marker 4302 for moving the calibrator 850 and the user's own hand 4301 are displayed together. Thus, the user can intuitively move the calibrator 850 to the marker 4302.

[0544] Next, in step S4207, the display device controller 801 determines whether a calibration processing completion notification has been received from the camera body 1. If the determination in step S4207 indicates that a completion notification has been received, the process ends. Conversely, if the determination in step S4207 indicates that a completion notification has not yet been received, the process returns to step S4203, and the steps starting from step S4203 are executed sequentially.

[0545] As described above, the calibration operation reduces the hassle for the user of orienting their face towards directions such as front, upper left, lower left, upper right, and lower right. This facilitates the calibration operation, i.e., the user's operation during calibration. Therefore, the convenience of using the camera body 1 and the calibrator 850 is improved.

[0546] Other embodiments

[0547] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0548] Although the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims is to be interpreted in the broadest sense to include all such modifications, equivalent structures, and functions.

[0549] This application claims priority to Japanese Patent Application No. 2021-125341, filed on July 30, 2021, and Japanese Patent Application No. 2022-038688, filed on March 11, 2022, the entire contents of which are incorporated herein by reference.

Claims

1. A camera device, comprising: An observation direction detection unit is suitable for wearing on a part of the user's body other than the head and is configured to detect the user's observation direction. At least one camera unit, which is adapted to be worn on a user's body and configured to capture images; A recording direction determination unit is configured to determine the recording direction using the detection result of the observation direction detection unit; A deviation detection unit is configured to detect deviations of the camera device relative to the user's body; as well as An image recording unit is configured to record a portion of an image captured by the camera unit in a recording area determined by a recording direction determined by the recording direction determination unit and a deviation detected by the deviation detection unit. The observation direction detection unit includes: An infrared irradiation unit is configured to irradiate an infrared irradiation surface, including the user's chin, with infrared light; and An infrared detection unit is configured to detect reflected light from the infrared light reflected by the infrared irradiation surface, and The observation direction detection unit outputs the user's observation direction in the horizontal direction as an angle in the first detection direction, and outputs the user's observation direction in the vertical direction as an angle in the second detection direction orthogonal to the first detection direction. The observation direction detection unit calculates the angle in the second detection direction based on the intensity of reflected light at the user's chin position.

2. The image pickup apparatus according to claim 1, wherein The deviation detection unit is configured to detect deviations of the camera device based on the deviations of the user's throat and chin positions relative to a reference position.

3. The apparatus according to claim 2, wherein The reference position is determined based on information related to calibration.

4. The camera device according to any one of claims 1 to 3 further includes an observation direction correction unit, the observation direction correction unit being configured to correct the output of the observation direction detection unit when the deviation detection unit detects a deviation of the camera device.

5. The camera device according to any one of claims 1 to 3, further comprising a recording image correction unit, the recording image correction unit being configured to correct an image to be recorded by the image recording unit when the deviation detection unit detects a deviation of the camera device.

6. The apparatus according to claim 5, wherein When the deviation detection unit detects a tilt deviation of the camera device relative to the user's body, the image recording correction unit is configured to perform tilt correction on the image to be recorded.

7. The apparatus according to claim 5, wherein When the deviation detection unit detects a deviation of the camera device, the image recording correction unit is configured to expand the recording area of ​​the image.

8. The camera device according to any one of claims 1 to 3, 6 and 7, further comprising a posture detection component configured to detect the user's posture. wherein If the output of the posture detection component changes by more than a predetermined threshold, the deviation detection unit is configured to stop detecting the deviation.

9. The camera device according to claim 8, further comprising: A tilt angle detection unit is configured to detect the tilt angle of the user's head; as well as A recorded image tilt correction unit is configured to perform tilt correction on the recorded image to be recorded by the image recording unit based on the detection result of the tilt angle detection unit.

10. The apparatus according to claim 9, wherein It is possible to set one of a first mode and a second mode. In the first mode, the recorded image tilt correction unit performs correction, and in the second mode, the recorded image tilt correction unit does not perform correction.

11. The image pickup apparatus according to claim 9 or 10, wherein If the tilt angle of the user's head detected by the tilt angle detection unit deviates from the angle of the vertical direction detected by the posture detection component by less than a first threshold, the recorded image tilt correction unit is configured to perform tilt correction on the recorded image based on the vertical direction. Wherein, when the angle deviation is not less than the first threshold, the recorded image tilt correction unit is configured to perform tilt correction on the recorded image based on the tilt angle of the user's head.

12. The image pickup apparatus according to claim 9 or 10, wherein The tilt angle detection unit includes a distance measuring unit configured to measure distance, and the tilt angle detection unit calculates the tilt angle of the user's head based on the distance to the area or point of the user's head measured by the distance measuring unit.

13. The apparatus according to claim 12, wherein The area or point on the head used to calculate the tilt angle of the head is arranged on both sides of a line connecting the user's throat position and chin position.

14. The camera device according to any one of claims 1 to 3, 6, 7, 9, 10 and 13, wherein, When the detection result vibrates at a frequency greater than a predetermined threshold, the recording direction determination unit is configured to determine the recording direction based on the upper peak point of the waveform of the detection result in the pitch direction of the observation direction.

15. The camera device of claim 14, further comprising a calibration unit configured to calibrate the observation direction detection unit. in, The calibration unit enables the user to perform calibration with their mouth closed.

16. The camera device according to claim 14, wherein, The predetermined threshold for the frequency is approximately 1 Hz to 10 Hz.

17. The camera device of claim 14, further comprising a sound detection component configured to detect the user's voice. in, When the sound detection component detects a sound greater than a predetermined threshold, the recording direction determination unit is configured to determine the recording direction based on the upper peak point of the waveform of the detection result in the yaw direction.

18. The camera device according to any one of claims 1 to 3, 6, 7, 9, 10, 13, 15, 16 and 17, wherein, The deviation detection unit is configured to use the detection result of the observation direction detection unit to detect deviation of the camera device.

19. The camera device according to any one of claims 1 to 3, 6, 7, 9, 10, 13, 15, 16 and 17, further comprising a warning component configured to warn a user if the deviation detection unit detects a deviation of the camera device.

20. The camera device according to claim 19, wherein, The warning component is configured to issue a warning if the deviation detection unit continuously detects a deviation of the camera device exceeding a predetermined time period.

21. The camera device according to claim 19, wherein, The warning component is configured to issue a warning if the deviation detection unit detects that the camera device deviates more than a predetermined number of times.

22. A calibration system, comprising: Camera equipment, including: The camera unit is configured to capture images. A facial orientation detection unit is configured to detect the facial orientation of a user using the camera device. An indicator detection unit is configured to detect indicators for locating a display device based on images captured by the camera unit. A calibration unit is configured to perform calibration when associating the facial orientation with the image, provided that the indicator detected by the indicator detection unit is at a predetermined position; and The display device includes: An image receiving unit is configured to receive images captured by the camera unit; A drawing unit configured to draw a marker indicating the mobile destination of the display device to a user, superimposed on an image received by the image receiving unit; and A receiving image display unit is configured to display an image of the mark drawn by the drawing unit. The facial orientation detection unit includes: An infrared irradiation unit is configured to irradiate an infrared irradiation surface, including the user's chin, with infrared light; and An infrared detection unit is configured to detect reflected light from the infrared light reflected by the infrared irradiation surface, and The facial orientation detection unit outputs the user's facial orientation in the horizontal direction as an angle in the first detection direction, and outputs the user's facial orientation in the vertical direction as an angle in the second detection direction orthogonal to the first detection direction. The facial orientation detection unit calculates the angle in the second detection direction based on the intensity of reflected light at the user's chin position.

23. The calibration system according to claim 22, wherein, The display device further includes an indicator display unit configured to display the indicator.

24. The calibration system according to claim 23, wherein, The indicator display unit is configured to change the shape or color of the indicator over time.

25. The calibration system according to any one of claims 22 to 24, wherein, The calibration unit is configured to calculate correction information that shows the relationship between the coordinates of the indicators detected by the indicator detection unit and the facial direction detected by the facial direction detection unit.

26. The calibration system according to claim 25, wherein, The camera device also includes a correction information storage unit, which is configured to store the correction information.

27. The calibration system according to claim 25, wherein, The camera device further includes a correction information sending unit, which is configured to send the correction information. The display device further includes: a calibration information receiving unit configured to receive the calibration information; and a calibration information storage unit configured to store the calibration information received by the calibration information receiving unit.

28. The calibration system according to any one of claims 22, 23, 24, 26 and 27, wherein, The camera device further includes a coordinate transmitting unit configured to transmit coordinate information related to the moving destination of the display device. The display device further includes a coordinate receiving unit configured to receive the coordinate information. The drawing unit draws a marker indicating the moving destination of the display device based on the coordinate information received by the coordinate receiving unit.

29. A control method for a camera device, the control method comprising: The observation direction detection step is used to detect the user's observation direction; The camera step is used to capture images using a camera unit worn on the user's body; A recording direction determination step is used to determine the recording direction using the detection result of the observation direction detection step; A deviation detection step is used to detect the deviation of the camera device relative to the user's body; as well as The image recording step is used to record a portion of the image captured by the camera unit in an area determined by the recording direction determined in the recording direction determination step and the deviation of the camera device detected in the deviation detection step. The observation direction detection step includes: Irradiate the infrared-illuminated area, including the user's chin, with infrared light. Detect the reflected light of the infrared light reflected by the infrared irradiated surface. Output the user's viewing direction in the horizontal direction as the angle in the first detection direction. The output shows the user's vertical viewing direction as an angle in a second detection direction orthogonal to the first detection direction, and... The angle in the second detection direction is calculated based on the intensity of the reflected light at the user's chin position.

30. A non-transitory computer-readable storage medium for storing a control program that causes a computer to perform the control method according to claim 29.

31. A computer program product comprising a control program that causes a computer to perform the control method according to claim 29.

Images