Tracking device, tracking method, and program
The tracking device and method use a line-of-sight acquisition unit to ensure the user's intended subject is tracked by limiting tracking based on viewpoint position changes, addressing the issue of multiple moving subjects and maintaining focus on the intended subject.
Patent Information
- Application Number
- JP2020157096
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-09-18
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-09-18
AI Technical Summary
Existing tracking technologies fail to accurately track a user's intended subject when multiple moving subjects are present, leading to the potential loss of the intended subject due to irregular changes in the user's viewpoint position.
A tracking device and method that utilize a line-of-sight acquisition unit to determine the user's viewpoint position, allowing tracking only when the change in viewpoint position is within a threshold and stopping tracking when the change exceeds this threshold, with additional methods to identify and display detection frames to assist the user in finding the intended subject.
Enables tracking control that aligns with the user's intention by preventing the device from tracking unintended subjects, ensuring the user can easily locate and maintain focus on the intended subject.
Smart Images

Figure 0007577490000008 
Figure 0007577490000009 
Figure 0007577490000010
Abstract
Description
[Technical field]
[0001] The present invention relates to a tracking device, a tracking method, and a program. [Background technology]
[0002] There is known a technology for detecting the position (viewpoint position) where a user is looking on the display unit of an electronic device. There is also a technology for tracking a specific subject in an image, for example, to keep the subject in focus for shooting. Patent Document 1 describes a technology for changing the position of the subject being tracked in an image displayed on the display unit to track another subject closer to the viewpoint position when there is a difference between the position of the subject being tracked in the image displayed on the display unit and the viewpoint position of the user. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Application Publication No. 5-53043 Summary of the Invention [Problem to be solved by the invention]
[0004] In the technology of Patent Document 1, if a plurality of subjects are moving in an image and the user loses sight of the subject that he or she wants to photograph, there is a possibility that an object unintended by the user will be tracked.
[0005] Therefore, an object of the present invention is to provide a tracking device that enables tracking control according to a user's intention. [Means for solving the problem]
[0006] One aspect of the present invention is a method for producing a composition comprising the steps of: An acquisition means for acquiring a viewpoint position, which is a position where a user is looking at the display means; a tracking means for tracking an object displayed on the display means; a control means for causing the tracking means to perform a tracking process for tracking an object corresponding to the viewpoint position when the degree of irregularity of the change in the viewpoint position is equal to or less than a first threshold, and for causing the tracking means to stop the tracking process when the degree of irregularity of the change in the viewpoint position is greater than the first threshold; The tracking device is characterized by having the following features.
[0007] One aspect of the present invention is a method for producing a composition comprising the steps of: A tracking method performed by a tracking device having a tracking means for tracking an object displayed on a display means, comprising: an acquisition step of acquiring a viewpoint position, which is a position where a user is looking at the display means; a control step of causing the tracking means to perform a tracking process for tracking an object corresponding to the viewpoint position when the degree of irregularity of the change in the viewpoint position is equal to or less than a first threshold, and causing the tracking means to stop the tracking process when the degree of irregularity of the change in the viewpoint position is greater than the first threshold; The tracking method is characterized by having the following features. Effect of the Invention
[0008] According to the present invention, it is possible to provide a tracking device that enables tracking control according to a user's intention. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a diagram illustrating the configuration of a digital camera according to a first embodiment. [Diagram 2] 3 is a diagram illustrating an example of the configuration of a line-of-sight acquiring unit according to the first embodiment. [Diagram 3] 3 is a diagram illustrating an example of the configuration of a line-of-sight acquiring unit according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing the internal configuration of an image processing unit according to the first embodiment. [Diagram 5] 4 is a flowchart of a tracking process according to the first embodiment. [Figure 6] 3A and 3B are diagrams for explaining a tracking frame or a detection frame according to the first embodiment. [Figure 7] 4 is a graph showing a power spectrum according to the first embodiment. [Figure 8] 4 is a graph showing a viewpoint trajectory according to the first embodiment. [Figure 9] 11 is a flowchart of an execution determination process according to the second embodiment. [Figure 10] FIG. 13 is a diagram illustrating a neural network according to a third embodiment. [Figure 11] 13 is a flowchart of an execution determination process according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention. In addition, multiple features can be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.
[0011] <Embodiment 1> [Explanation of the digital camera configuration] 1 is a diagram showing the configuration of a digital camera 100 (imaging device) according to embodiment 1. Note that instead of the digital camera 100, a tracking device (electronic device) capable of tracking a subject (object), such as a smartphone or a PC (computer) having each of the components of the digital camera 100, may be used.
[0012] In FIG. 1, the lens unit 150 is a lens unit equipped with an interchangeable photographing lens. The lens 103 usually has multiple lenses, but only one lens is shown here. The communication terminal 6 is a terminal through which the lens unit 150 communicates with the digital camera 100. The communication terminal 10 is a terminal through which the digital camera 100 communicates with the lens unit 150. The lens unit 150 communicates with the system control unit 50 via the communication terminals 6 and 10. The lens unit 150 controls the aperture 102 via the aperture drive circuit 2 by the internal lens system control circuit 4. The lens unit 150 also adjusts the focus by displacing the position of the lens 103 via the AF drive circuit 3. The lens system control circuit 4 stores information about the lens (such as information about the focal length).
[0013] The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50 .
[0014] The imaging unit 22 captures an image of a subject through the aperture 102 and the lens 103. The imaging unit 22 is an imaging element including a CCD or CMOS element that converts an optical image into an electrical signal. In the imaging unit 22, pixels in which a photoelectric conversion unit is divided into a plurality of parts are provided for one microlens. As a result, light is divided and incident on each pixel, so that a phase difference detection signal can be obtained from the photoelectric conversion unit. In addition, the imaging unit 22 can obtain an imaging signal by adding signals from each pixel. Such a pixel can serve as both a focus detection pixel and an imaging pixel. Note that the imaging unit 22 may be provided with only pixels for imaging, and in this case, focus detection may be realized by a contrast method. In this way, the signal acquired by the imaging unit 22 can be used not only for imaging, but also for exposure control and focus detection control.
[0015] The A / D converter 23 converts the analog signal output from the imaging unit 22 into a digital signal. The A / D converter 23 outputs the converted digital signal as an image (video) to the image processing unit 24 or The data is output to the memory control unit 15 or the like.
[0016] The image processing unit 24 performs resizing (predetermined pixel interpolation and reduction processing) and color conversion processing on the image (data) output from the A / D converter 23 or the image output from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image, and the system control unit 50 performs exposure control and distance measurement control based on the results of the arithmetic processing. This realizes TTL (through-the-lens) type AF (autofocus) processing, AE (autoexposure) processing, and EF (flash pre-flash) processing. Furthermore, the image processing unit 24 performs predetermined arithmetic processing using the captured image, and performs TTL type AWB (auto white balance) processing based on the obtained arithmetic results.
[0017] In this embodiment, the image processing unit 24 can perform subject detection processing and tracking processing based on an image (video; moving image). The internal configuration of the image processing unit 24 for performing subject detection processing and tracking processing will be described later with reference to FIG.
[0018] The image (output data) output from the A / D converter 23 is written into the memory 32 via the image processing unit 24 and the memory control unit 15, or via the memory control unit 15. The memory 32 stores images obtained by the imaging unit 22 and converted into digital signals by the A / D converter 23, and images to be displayed on the display unit 28. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, and moving images and audio for a predetermined period of time.
[0019] The memory 32 also serves as a memory for image display (video memory). The D / A converter 19 converts the digital signal of the image for image display stored in the memory 32 into an analog signal and supplies it to the display unit 28. The image for display written in the memory 32 in this manner is supplied via the D / A converter 19 and displayed by the display unit 28.
[0020] The display unit 28 performs display on a display device such as an LCD according to the analog signal acquired from the D / A converter 19. When the digital signals stored in the memory 32 are converted to analog signals by the D / A converter 19 and sequentially transferred to the display unit 28, the display unit 28 performs live view display (LV). Hereinafter, an image displayed in live view display is referred to as a live view image (LV image). The subject currently being imaged by the imaging unit 22 appears in the live view image.
[0021] Display unit 28 may be an electronic viewfinder seen through an eyepiece (not shown), or a display provided on the back of digital camera 100. Display unit 28 may also have both an electronic viewfinder and a display on the back.
[0022] The non-volatile memory 56 is an electrically erasable and recordable memory. For example, an EEPROM or the like is used for the non-volatile memory 56. The non-volatile memory 56 stores constants and programs for the operation of the system control unit 50. For example, the non-volatile memory 56 stores programs for executing various flowcharts described later in this embodiment.
[0023] The system control unit 50 controls each component of the digital camera 100 by executing a program stored in the nonvolatile memory 56. A RAM is used for the system memory 52. The system control unit 50 can load constants and variables for the operation of the system control unit 50, and programs read from the nonvolatile memory 56, into the system memory 52. The system control unit 50 also performs display control by controlling the memory 32, the D / A converter 19, the display unit 28, etc. The system timer 53 is a timing unit that measures the time used for various controls and the time of a built-in clock.
[0024] The power supply control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to which electricity is applied, and the like. The power supply control unit 80 detects whether a battery (power supply unit 30) is attached, the type of battery, and the remaining battery level. Based on the detection result and instructions from the system control unit 50, the power supply control unit 80 controls the DC-DC converter to supply power from the power supply unit 30 to each component (including the recording medium 200). The power supply unit 30 includes a primary battery (such as an alkaline manganese battery or a Li battery), a secondary battery (such as a NiCd battery, a NiMH battery, or a Li battery), an AC adapter, and the like.
[0025] The recording medium interface 18 (I / F) is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images. The recording medium 200 includes a semiconductor memory, a magnetic disk, or the like.
[0026] The communication unit 54 is connected to the outside through wireless communication or wired communication, and transmits and receives video signals and audio signals. The communication unit 54 can be connected to a network such as an intranet or the Internet. The communication unit 54 can transmit images (including live view images) captured by the imaging unit 22 and images recorded on the recording medium 200. The communication unit 54 can also receive images and various other information from external devices.
[0027] The attitude detection unit 55 detects the attitude of the digital camera 100 with respect to the direction of gravity. Based on the attitude detected by the attitude detection unit 55, it is possible to determine whether the image captured by the imaging unit 22 was captured with the digital camera 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 55 to the image file of the image captured by the imaging unit 22, or rotate and record the image. An acceleration sensor, a gyro sensor, or the like can be used as the attitude detection unit 55. By using the acceleration sensor or gyro sensor as the attitude detection unit 55, it is also possible to detect the movement of the digital camera 100 (panning, tilting, lifting, whether it is stationary, etc.). In addition, the attitude detection unit 55 can detect the rotation angle γ (yaw angle) of the digital camera 100 about the z-axis in an xyz space with the z-axis direction being the direction of gravity. The attitude detection unit 55 can also detect the rotation angle β in the up-down direction of the digital camera 100 (the pitch angle around the y-axis along the left-right direction of the digital camera 100). The attitude detection unit 55 can also detect the rotation angle α in the left-right tilt direction of the digital camera 100 (the roll angle around the x-axis along the front-rear direction of the digital camera 100).
[0028] The operation unit 70 has a plurality of operation members that accept operations from the user. The operation unit 70 has buttons (menu button and SET button) and a four-way key as operation members to select a menu, select a mode, play back captured video images, and the like. For example, when the menu button is pressed, a menu screen in which various settings can be made is displayed on the display unit 28. The user can intuitively make various settings using the menu screen displayed on the display unit 28 and the four-way key and SET button.
[0029] Moreover, each operation member of the operation unit 70 can operate as a button (function button) for executing a function corresponding to a functional icon by selecting the functional icon on the screen displayed on the display unit 28. The function buttons include, for example, an end button, a back button, an image forward button, a jump button, a narrowing down button, and an attribute change button.
[0030] Furthermore, the operation unit 70 includes a mode changeover switch 60, a shutter button 61, and a power switch 72 as operation members.
[0031] The mode changeover switch 60 changes the operation mode of the system control unit 50 between a still image shooting mode and a motion image shooting mode. The mode is switched to one of the still image shooting mode, playback mode, etc. Modes included in the still image shooting mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). There are also various scene modes and custom modes that are shooting settings according to the shooting scene. The user can directly switch to one of these modes using the mode changeover switch 60. Alternatively, after switching once to a list screen of shooting modes with the mode changeover switch 60, the user may selectively switch to one of the displayed modes using another operating member. Similarly, the video shooting mode may also include a plurality of modes.
[0032] The shutter button 61 includes a first shutter switch 62 and a second shutter switch 64. When the first shutter switch 62 is turned ON by half-pressing (instruction to prepare for shooting) during the operation of the shutter button 61, a first shutter switch signal SW1 is generated. In response to the generation of the first shutter switch signal SW1, the system control unit 50 starts shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing.
[0033] When the second shutter switch 64 is turned ON by completing the operation of the shutter button 61, that is, by fully pressing it (photographing instruction), a second shutter switch signal SW2 is generated. Upon generation of the second shutter switch signal SW2, the system control unit 50 starts a series of operations for photographing processing, from reading out a signal from the imaging unit 22 to writing the captured image to the recording medium 200 as an image file.
[0034] The power switch 72 is an operating member for switching the power of the digital camera 100 between ON and OFF.
[0035] The line-of-sight acquisition unit 701 detects (acquires) the position (viewpoint position; line-of-sight position) at which the user is looking on the display unit 28. Fig. 2 shows an example of the line-of-sight acquisition unit 701. The line-of-sight acquisition unit 701 has an image sensor 701a, a light receiving lens 701b, a dichroic mirror 701c, an eyepiece 701d, an illumination light source 701e, and a control unit (not shown). At this time, a live view image captured via the lens unit 150 is displayed on the display unit 28.
[0036] In order to detect the viewpoint position of the user who is looking at the display unit 28 through the eyepiece lens 701d, first, the illumination light source 701e projects infrared light onto the eyeball 301. Then, the infrared light reflected by the eyeball 301 is further reflected by the dichroic mirror 701c, passes through the light receiving lens 701b, and enters the image sensor 701a. The image sensor 701a photographs the eyeball 301 with the incident infrared light to obtain an eyeball image. The control unit of the line of sight acquisition unit 701 extracts the area of the pupil from the photographed eyeball image. The control unit detects the rotation angle of the optical axis of the eyeball 301 of the user looking into the viewfinder field, and detects the user's line of sight from the detected rotation angle. Then, the control unit detects a position (area) on the display unit 28 corresponding to the user's line of sight (direction of the eyes; direction of looking with the eyes) as the viewpoint position. Note that the line of sight acquisition unit 701 may photograph one or both of the user's eyes.
[0037] FIG. 3 shows an example of a line-of-sight acquisition unit 701 different from that shown in FIG. 2. Here, the line-of-sight acquisition unit 701 has a camera 701f, an illumination light source 701e, and a control unit. In FIG. 3, a live view image captured through the lens unit 150 is displayed on the display unit 28. In FIG. 3, a camera 701f is provided on the rear of the digital camera 100 to capture an image of the face 300 (eyeballs 301, 302) of the user observing the display unit 28. In FIG. 3, the range that the camera 701f can capture is indicated by a dashed line. When light is projected from the illumination light source 701e onto the user's face, the camera 701f captures an eyeball image (face image). Then, the control of the line-of-sight acquisition unit 701 is The unit detects the user's line of sight and viewpoint position from the eyeball image.
[0038] 2 and 3, the eyeball image may be converted into a digital signal by an A / D converter (not shown) and transmitted to the system control unit 50. In this case, the system control unit 50 detects the viewpoint position based on the eyeball image. For this reason, the gaze acquisition unit 701 is not limited to the configurations shown in FIGS. 2 and 3, and may have any configuration as long as it can acquire information for detecting the user's gaze (viewpoint position).
[0039] (Image processing unit) Hereinafter, a part of the internal configuration of the image processing unit 24 for detecting and tracking a subject will be described with reference to the configuration diagram of Fig. 4. The image processing unit 24 has an image acquisition unit 410, a detection unit 411, and a tracking unit 412.
[0040] The image acquisition unit 410 acquires an image (live view image) from the A / D converter 23. The image acquisition unit 410 is not limited to acquiring an image from the A / D converter 23, and may acquire an image from an external device of the digital camera 100.
[0041] The detection unit 411 detects a subject (object) included in the video acquired by the image acquisition unit 410. The subject detected by the detection unit 411 is a type of subject (object) that is likely to be photographed by a user, such as a person, an animal, or a vehicle. The subject detected by the detection unit 411 may be a type of subject selected in advance by a user. The detection unit 411 can detect the subject by using an existing method such as a convolutional neural network.
[0042] Tracking unit 412 tracks a specific subject in the image displayed on display unit 28. Tracking unit 412 can also obtain the position (x coordinate and y coordinate) of the subject being tracked in the image. System control unit 50 can switch between starting and stopping tracking of the subject by tracking unit 412.
[0043] [Tracking process] Hereinafter, a process (tracking process; tracking method; tracking device control method) for controlling the tracking unit 412 so that it can track a subject desired by the user will be described with reference to the flowchart of Fig. 5. The process of this flowchart is realized by the system control unit 50 loading a program recorded in the non-volatile memory 56 into the system memory 52 and executing it. Also, before the start of this flowchart, it is assumed that the tracking unit 412 is tracking one subject in the video. At this time, a tracking frame showing the subject being tracked is displayed on the display unit 28.
[0044] In S501, the system control unit 50 controls the image acquisition unit 410 to acquire a video (live view image) from the A / D converter 23. Note that the image acquisition unit 410 may acquire the video from the memory 32 or the like, instead of acquiring the video from the A / D converter 23.
[0045] In S502 , the system control unit 50 acquires the user's viewpoint position from the line-of-sight acquisition unit 701 .
[0046] In S503, system control unit 50 controls tracking unit 412 to acquire the position (display position; position information) of the subject (tracked subject) being tracked by tracking unit 412 in the video acquired by image acquisition unit 410. Note that, when tracking unit 412 stops tracking the subject, detection unit 411 acquires the current position of the subject that tracking unit 412 was tracking at the time tracking was stopped.
[0047] In S504, the system control unit 50 acquires the degree of irregularity of the trajectory of the viewpoint position (line of sight) (viewpoint trajectory; change in viewpoint position; change in line of sight). A method of acquiring the degree of irregularity of the viewpoint trajectory will be described later.
[0048] In S505, the system control unit 50 judges whether the irregularity of the viewpoint trajectory is equal to or less than a predetermined threshold THr. Here, if the irregularity of the viewpoint trajectory is equal to or less than the threshold THr, the system control unit 50 can judge that the viewpoint position is changing regularly and the user is tracking the subject with the line of sight. On the other hand, if the irregularity of the viewpoint trajectory is greater than the threshold THr, the system control unit 50 can judge that the viewpoint position is changing irregularly and the user is not tracking the subject (has lost sight of it). If the irregularity of the viewpoint trajectory is equal to or less than the threshold THr, the process proceeds to S508. If the irregularity of the viewpoint trajectory is greater than the threshold THr, the process proceeds to S506.
[0049] In S506, the system control unit 50 controls the tracking unit 412 to stop tracking by the tracking unit 412. Then, the system control unit 50 stops displaying the tracking frame on the display unit 28. Note that if tracking by the tracking unit 412 has already been stopped, the system control unit 50 does not execute any process in S506.
[0050] In S507, the system control unit 50 controls the detection unit 411 to detect one or more subjects from the video, and displays frames (detection frames) showing each of these subjects on the display unit 28. The system control unit 50 does not need to display the detection frames, and may simply display (highlight) the positions of one or more subjects in a way that makes it easy for the user to grasp them. This makes it easier to grasp the positions of the subjects included in the video, so that the user can easily track the desired subject again with his or her line of sight. At this time, the system control unit 50 may notify (notify) the user by sound or image that the viewpoint trajectory is irregular. When the process of S507 ends, the processes of S501 to S505 are performed again. That is, the system control unit 50 controls so that the tracking suspension state continues and the detection frames showing each subject continue to be displayed until the viewpoint trajectory becomes regular.
[0051] In S508, the system control unit 50 judges whether the difference DF between the position of the subject acquired in S503 and the viewpoint position acquired in S502 is equal to or less than a predetermined threshold THp. Here, if the difference DF is equal to or less than the threshold THp, the system control unit 50 can judge that the user is following the subject whose position was acquired in S503 with his / her line of sight. On the other hand, if the difference DF is greater than the threshold THp, the system control unit 50 can judge that the user is following a subject other than the subject whose position was acquired in S503 with his / her line of sight. If the difference DF is equal to or less than the threshold THp, the process proceeds to S509. If the difference DF is greater than the threshold THp, the process proceeds to S510.
[0052] In S509, the system control unit 50 controls the tracking unit 412 to track (continue to track) the subject whose position was acquired in S503.
[0053] In S510, the system control unit 50 can determine that the user is tracking the subject closest (close) to the viewpoint position, and therefore controls the tracking unit 412 to track the subject closest (close subject) to the viewpoint position (change the tracked subject).
[0054] In S511, the system control unit 50 displays on the display unit 28 a tracking frame that indicates the subject being tracked by the tracking unit 412.
[0055] In this way, in S509 and S510, the system control unit 50 determines that the subject corresponding to the viewpoint position is the main subject (the subject that the user wants to track (photograph)), and A tracking frame showing the subject corresponding to the viewpoint position is displayed on the display unit 28.
[0056] In S512, the system control unit 50 selects the focus detection area closest to the tracking frame, and acquires (detects) the focus state (defocus amount and direction) using the signal (phase difference detection signal) acquired by the imaging unit 22.
[0057] In S513, the system control unit 50 calculates the lens drive amount and drive direction corresponding to the defocus amount and defocus direction acquired in S512. The system control unit 50 adjusts the focal distance by controlling the position of the lens 103 according to the calculated lens drive amount and drive direction, and controls the imaging unit 22 to perform shooting.
[0058] In this manner, in this embodiment, whether or not the user is tracking the subject with his / her line of sight is determined based on the irregularity of the viewpoint trajectory, and tracking by tracking unit 412 is controlled according to the result. If the degree of irregularity of the viewpoint trajectory is high (if digital camera 100 determines that the user is not tracking the subject with his / her line of sight), digital camera 100 stops tracking the subject by tracking unit 412 and displays detection frames indicating each subject included in the video. This makes it possible to reduce the possibility of digital camera 100 tracking a subject that does not match the user's intention, thereby enabling tracking control according to the user's intention.
[0059] [Tracking frame, detection frame display] 6(A) and 6(B) are diagrams for explaining the display of a tracking frame or detection frame on display unit 28. Here, it is assumed that digital camera 100 is shooting a sports scene. FIGS. 6(A) and 6(B) show a scene in which person 601 and / or person 603 are playing a sport using a ball 602. Detection frames 621-623 are display items that indicate subjects displayed in the video. Tracking frame 611 is a display item that indicates a subject that digital camera 100 is tracking.
[0060] In FIG. 6A, the viewpoint trajectory 600 is regular (S505 YES), and the distance between the viewpoint position and the person 601, who is the subject, is close (S508 YES). For this reason, it is considered that the subject indicated by the tracking frame 611 and the subject (main subject) that the user wants to photograph coincide with the person 601. At this time, in this embodiment, the tracking frame 611 is displayed so as to track the person 601 (S509, S511), so that the user can easily continue to track the person 601, who is the main subject, by looking at the tracking frame 611. At this time, the detection frame is not displayed so that the user's gaze does not become scattered.
[0061] On the other hand, in FIG. 6(B), the viewpoint position moves randomly, and viewpoint trajectory 600 is highly irregular (S505 NO). In this case, it is considered that the user has lost sight of the subject (main subject) that he / she wants to photograph. For this reason, digital camera 100 stops displaying tracking frame 611 (S506) and displays detection frames 621-623 that indicate subjects included in the video (S507) so that the user can easily find the subject (main subject) that he / she wants to photograph. This allows the user to find the main subject from the subjects indicated by detection frames 621-623, making it easier for the user to find the main subject than when the detection frames are not displayed.
[0062] Furthermore, since the system control unit 50 only needs to enable the user to grasp the position of the subject, it may display any display item, such as a star-shaped or circular mark, at the position of the subject instead of the detection frame or tracking frame. Note that the system control unit 50 may change the color or thickness of the tracking frame and detection frame displayed on the display unit 28. The system control unit 50 may also differentiate the display of the tracking frame and detection frame by other methods. This makes it easier for the user to recognize whether the detection frame or the tracking frame is displayed, making it easier for the user to find the main subject. It will become easier.
[0063] [Viewpoint trajectory acquisition process] The process (acquisition method; calculation method) by which the system control unit 50 acquires the irregularity of the viewpoint trajectory (change in viewpoint position) will be described in detail below.
[0064] (When frequency is used to obtain irregularity) First, a case will be described in which frequency information in the time axis direction of the viewpoint trajectory is used to obtain the irregularity of the viewpoint trajectory. If time is t and the coordinates of the viewpoint position at each time are (x(t), y(t)), X(ω) which is the result of Fourier transform of x(t) and Y(ω) which is the result of Fourier transform of y(t) are expressed by the following formulas 1 and 2.
number
[0065] Here, Δt is the time interval for acquiring the viewpoint position, and frequency ω is the frequency in the period from time t0 to time t0+NΔt. As shown in Equation 3, the power spectrum P(ω) is defined as the sum of the square of the absolute value of X(ω) and the square of the absolute value of Y(ω).
number
[0066] 7, the horizontal axis represents frequency ω, and the vertical axis represents power spectrum P(ω). The system control unit 50 calculates the sum S of power spectrum P(ω) at or above a predetermined frequency ω0 as the degree of irregularity. Then, in S505, the system control unit 50 determines whether the sum S, which is the degree of irregularity, is equal to or less than a threshold value.
[0067] (When using autocorrelation as the measure of irregularity) Next, an example will be shown in which a value based on the autocorrelation of the viewpoint trajectory is used as the degree of irregularity. Fig. 8 is a graph in which x(t), which is the x position (x coordinate) of the viewpoint trajectory at time t, is plotted against time t. In adjacent time spans 801 and 802, if the viewpoint trajectory is regular, the autocorrelation approaches 1, and if it is random, the autocorrelation approaches 0.
[0068] The absolute value R(T) of the autocorrelation can be expressed as in Equation 4. Here, Δt is the time interval for acquiring the line of sight, and T=NΔt is the time width for calculating the correlation.
number
[0069] In addition, in Equation 4, the autocorrelation is taken in adjacent sections of time width T, but the sections do not necessarily have to be adjacent as long as they are close to each other. Also, instead of simply adding the left side (x component) and the right side (y component) of Equation 4, the left side and the right side may be weighted and added. In addition, in S505, for example, the system control unit 50 determines whether the irregularity of the viewpoint trajectory is equal to or less than a threshold value by taking the reciprocal of the absolute value R(T) of the autocorrelation as the irregularity of the viewpoint trajectory.
[0070] (When using other information for irregularity) Alternatively, the representative value (average value, mode or median) of the magnitude of the gaze velocity vector (vx, vy)=(dx / dt, dy / dt) or the magnitude of the acceleration vector (dvx / dt, vy / dt) over a time span T may be used as the irregularity. Here, d / dt represents the derivative with respect to time t. Furthermore, the irregularity of the gaze trajectory may be determined by any combination of the gaze vector, acceleration vector, autocorrelation, frequency, etc.
[0071] As described above, according to this embodiment, tracking in digital camera 100 is controlled based on the irregularity of the viewpoint trajectory. This enables tracking in digital camera 100 according to the user's intention.
[0072] [Variation 1] In addition to the viewpoint trajectory, the trajectory information of the position (display position) of the subject detected by the detection unit 411 can be used to determine the irregularity of the viewpoint trajectory. For example, the system control unit 50 calculates the cross-correlation between the vector of the viewpoint trajectory and the vector of the position trajectory of the subject in a certain time span T=NΔt. Then, the system control unit 50 determines the irregularity of the viewpoint trajectory as a value based on the absolute value of the cross-correlation between the vector of the viewpoint trajectory and all the vectors of the position trajectory of the subject detected by the detection unit 411 (for example, the reciprocal of the absolute value). For example, the reciprocal of the largest value of the absolute value of the cross-correlation between the vectors of the subject position trajectory and the vector of the viewpoint trajectory detected by the detection unit 411 is determined as the irregularity of the viewpoint trajectory. That is, in this modified example, if the viewpoint position and the subject position change in the same way, the viewpoint trajectory is determined to be a regular trajectory following the trajectory of the subject position. Here, the calculation method of the cross-correlation R(t0) in the period from time t0 to time t0+NΔt is expressed by the following formula 5.
number
[0073] Here, (x e (t), y e (t)) represents the viewpoint position at time t. (x о (t), y о R(t0) represents the position of the subject at time t. The cross-correlation R(t0) may also be calculated by weighting the x component (left side) and the y component (right side). The cross-correlation between the motion vector (velocity vector or acceleration vector) of the viewpoint position and the motion vector of the detected subject may also be used.
[0074] <Embodiment 2> In the second embodiment, a digital camera 100 is described that switches whether or not to perform the process (tracking control process of S504 to S507) for determining whether or not to perform tracking, described in the first embodiment, depending on the trajectory (change) of the position of a subject included in the video. Furthermore, the digital camera 100 according to this embodiment changes the threshold value THr for determining irregularity of the viewpoint trajectory in S505, depending on the trajectory (change) of the position of a subject included in the video.
[0075] For example, in a scene where a subject is moving regularly, it is easy for the user to continue to track the subject, so there is a high possibility that the user is tracking the subject indicated by the tracking frame with his / her line of sight. Therefore, it is highly likely that it is appropriate for the tracking unit 412 to continue tracking, and there is little need to perform the tracking control process in the first embodiment.
[0076] In addition, in scenes where multiple subjects are moving in various directions (such as sports scenes), Even when the user's viewpoint is tracking the subject appropriately, the viewpoint trajectory tends to be highly irregular. Therefore, in such a case, even if the user can track the subject moving in various directions with his / her line of sight, the viewpoint position moves irregularly, so it is erroneously determined that the user cannot track the subject with his / her line of sight. Therefore, in such a case, it is better that the threshold value THr for determining the irregularity of the viewpoint trajectory is larger than in other cases.
[0077] [Execution decision process] The process of determining whether or not to execute the tracking control process (execution decision process) will be described with reference to the flowchart shown in Fig. 9. Note that the description of the same processes as those in the first embodiment will be omitted. Note that when multiple subjects are included in the video, the "irregularity of subject trajectory" below will be read as "a representative value (average value, median value, mode, etc.) of the irregularity of the position trajectories (subject trajectories) of multiple subjects."
[0078] In S901, the system control unit 50 acquires (calculates) the irregularity of the subject trajectory and determines whether the irregularity of the subject trajectory is equal to or greater than a first threshold THr1. To determine the irregularity of the subject trajectory, the frequency of the subject trajectory, the autocorrelation in the time direction of the subject trajectory, the velocity vector, and the acceleration vector can be used, as with the viewpoint trajectory in the first embodiment. If the irregularity of the subject trajectory is equal to or greater than the first threshold THr1 (above threshold), the process proceeds to S902. If the irregularity of the subject trajectory is less than the first threshold THr1 (below threshold), the process proceeds to S905.
[0079] In S902, the system control unit 50 determines whether the irregularity of the subject trajectory is less than a second threshold THr2. Here, the second threshold THr2 is a value greater than the first threshold THr1. If the irregularity of the subject trajectory is less than the second threshold THr2, the process proceeds to S903. If the irregularity of the subject trajectory is equal to or greater than the second threshold THr2, the process proceeds to S904.
[0080] In S903, the system control unit 50 executes the processes of S504 to S513 including the tracking control process (S504 to S507). After the process of S507 is performed, the process proceeds to S501 in this flowchart.
[0081] In S904, the system control unit 50 increases the threshold value THr for the determination in S505 compared to when S902 is YES (when the irregularity of the subject trajectory is equal to or greater than the first threshold value THr1 and less than the second threshold value THr2). In this way, when the irregularity of the subject trajectory is high, the system control unit 50 increases the threshold value THr, which is the determination threshold value for the irregularity of the viewpoint trajectory. This reduces the possibility that, in a scene where the subject moves irregularly, it is determined that the subject is not tracked by the user's gaze (S505No) even though the subject is tracked by the user's gaze.
[0082] In S905, the system control unit 50 does not execute the tracking control process (S504 to S507). This is because, if the process has proceeded to S905, the trajectory of the subject's position is very regular, so it is easy to track the subject with the user's line of sight, and it is unlikely that the user is unable to track the subject. Note that the system control unit 50 executes the processes of S508 to S513 other than the processes of S504 to S507, which are the tracking control process.
[0083] As described above, in this embodiment, digital camera 100 switches whether to execute the tracking control process (S504 to S507) and the threshold value Thr for the irregularity of the viewpoint trajectory according to the subject trajectory. This makes it possible to reduce the amount of unnecessary processing due to the tracking control process and appropriately determine whether the user is tracking the subject with their line of sight.
[0084] <Embodiment 3> In this embodiment, when it is determined in S505 in the first embodiment that the degree of irregularity of the viewpoint trajectory is high, the system control unit 50 uses a machine learning device included in the digital camera 100 to determine a main subject (a subject that the user wishes to photograph (track)). Then, the system control unit 50 tracks the determined main subject and displays a tracking frame. Note that this is not limited to the above, and even when the viewpoint trajectory is regular (without going through the determination of S505), the system control unit 50 may determine the main subject using the machine learning device.
[0085] Below, a case will be described in which a neural network, which is one method of machine learning, is used in a machine learning device, but it is also possible to use other regression methods such as linear (non-linear) regression.
[0086] Fig. 10 shows an example of the structure of a neural network. The neural network shown in Fig. 10 includes an input layer 1001, a hidden layer 1002, an output layer 1003, and neurons 1004. Also, connection lines 1005 indicate the connections between the neurons 1004. Here, for convenience of illustration, only representative neurons and connection lines are numbered. The system control unit 50 inputs data to the input layer 1001 of the neural network and acquires data from the output layer 1003.
[0087] The number of neurons 1004 in input layer 1001 is the same as the dimension of the input data. The data input to input layer 1001 includes data on the viewpoint trajectory and the subject's position trajectory. In addition, output layer 1003 outputs two values, the x coordinate and the y coordinate of the main subject in the video. Therefore, the number of neurons 1004 in output layer 1003 is two. System control unit 50 can determine the main subject by outputting the two values, the x coordinate and the y coordinate of the main subject.
[0088] A connection line 1005 connecting the i-th neuron 1004 in the input layer 1001 and the j-th neuron 1004 in the hidden layer 1002 has a weight w ji The value z j can be calculated using the following Equation 6.
number
[0089] In Equation 6, x i represents a value input to the i-th neuron 1004 of the input layer 1001. If the number of neurons 1004 in the input layer 1001 connected to the j-th neuron 1004 in the hidden layer 1002 is N, i takes a value from 1 to N. j is called the bias, and is a parameter that controls the likelihood of the jth neuron 1004 firing.
[0090] Moreover, the function h(p) shown in Equation 6 and Equation 7 is a function that outputs the larger of p and 0. In other words, the function h(p) is a function that always outputs 0 when the input value p to the function is 0 or less, and outputs p, the same as the input value, when the input value p is greater than 0. The function h(p) is an activation function called ReLU (Rectified Linear Unit). It is also possible to use another function, such as a sigmoid function, as the activation function.
[0091] In addition, the value y k can be calculated by the following formula 8. Here, k in formula 8 is either 1 or 2. k=1 is represents a neuron 1004 that outputs the value of the x coordinate of the main subject, and k=2 represents a neuron 1004 that outputs the value of the y coordinate of the main subject.
number
[0092] In formula 8, z j represents the value output by the j-th neuron 1004 in the hidden layer 1002. If the number of all neurons 1004 in the hidden layer 1002 connected to the k-th neuron 1004 in the output layer 1003 is M, j takes values from 1 to M.
[0093] The function f is assumed to be an identity map. Since the coordinate values of the image are always positive, the ReLU function used in Equation 7 may be used for the function f. In addition, since only coordinate values normalized to [0, 1] are handled in this embodiment as described later, a sigmoid function may be used for the function f.
[0094] Here, in order to train the neural network, input data and correct answer data are prepared as training data. For example, the viewpoint trajectory when a person views a plurality of scenes is recorded in advance, and the viewpoint trajectory and the trajectory of the subject's position at that time are input as input data to the input layer 1001. The training data may be artificially padded by data augmentation technology. Each of the recorded viewpoint trajectories is normalized in advance to the range of [0,1] according to the width and height of the video in order to eliminate the influence of the size of the video (image). Moreover, the correct answer data as the training data indicates the normalized correct answer coordinates (x coordinate and y coordinate) on the image of the main subject. As the scene to be trained, a scene in which irregularities in viewpoint regulation may be detected (such as a scene in which multiple subjects move while intersecting) is preferable, but a scene in which tracking by gaze is easy may also be added.
[0095] During learning, all weights and biases are optimized to minimize the loss function L, which represents the degree of discrepancy between the coordinates output based on the input data (the trajectory of the viewpoint position and the trajectory of the subject position) and the correct coordinates. The loss function L can be a function of the sum of squares error as expressed in the following formula 9.
number
[0096] In Equation 9, the subscript k represents the coordinate components, with k=1 representing the x-coordinate and k=2 representing the y-coordinate. k is the normalized coordinate value output from the neuron 1004 in the output layer 1003. k is the normalized correct coordinate value of the main subject. By optimizing based on Equation 9, it is possible to determine the weights and biases so that the correct coordinate values and the output coordinate values are close to each other.
[0097] The loss factor L may be any function that indicates the degree of mismatch (degree of match) between the output coordinate value and the correct coordinate.
[0098] The learned weights and bias values are saved in advance in non-volatile memory 56, and are stored in memory 32 as necessary. As described above, using the learned weights and biases, the neural network outputs the normalized coordinate values (y1, y2) of the main subject based on Equations 5 to 8.
[0099] As described above, according to this embodiment, the trajectory of the viewpoint and the trajectory of the subject position are used. If the main subject can be accurately determined, digital camera 100 can track (photograph) the subject desired by the user.
[0100] <Embodiment 4> When the user frequently moves the position of digital camera 100, there is a possibility that the user may lose sight of the main subject. In addition, the user is likely to lose sight of the subject immediately after panning or when shooting with a telephoto lens. Therefore, in this embodiment, digital camera 100 switches between whether or not to execute the tracking control process (processing of S504 to S507) based on the movement information of digital camera 100 and the lens information.
[0101] [Execution decision process] The process of determining whether or not to execute the tracking control process (execution decision process) will be described with reference to the flowchart shown in Fig. 11. Note that the description of the same processes as those in the first and second embodiments will be omitted.
[0102] In S1101, the system control unit 50 determines whether the amount of movement of the digital camera 100 detected by the orientation detection unit 55 is equal to or greater than the threshold value THcam. If the amount of movement of the digital camera 100 is equal to or greater than the threshold value THcam, the process proceeds to S1102. If the amount of movement of the digital camera 100 is less than the threshold value THcam, the process proceeds to S1104.
[0103] Here, the amount of movement of the digital camera 100 can be a change in the angle of the digital camera 100, such as a pitch angle, a yaw angle, or a roll angle, which can be detected by the orientation detection unit 55. The amount of movement of the digital camera 100 can also be the amount of movement (shift movement amount) of the digital camera 100 in the horizontal direction, vertical direction, diagonal direction, or the like. Also, instead of the amount of movement of the digital camera 100, the number of times the amount of movement exceeds a predetermined threshold within a certain period of time can be used. It can be said that the more times the amount of movement exceeds the predetermined threshold, the more frequently the user moves the digital camera 100. Therefore, in this case, in S1101, the system control unit 50 determines whether the number of times the amount of movement of the digital camera 100 detected by the orientation detection unit 55 exceeds the predetermined threshold is equal to or greater than the threshold THcam.
[0104] In S1102, the system control unit 50 obtains information on the focal length of the lens from the lens system control circuit 4, and determines whether the focal length is equal to or greater than the threshold value THd. If the focal length is equal to or greater than the threshold value THd, the process proceeds to S1103. If the focal length is less than the threshold value THd, the process proceeds to S1104.
[0105] In S1103, the system control unit 50 executes the processes of S504 to S513, including the tracking control process (S504 to S507) of the first embodiment. If the process proceeds to S1103, it is considered that a telephoto lens is used due to the large focal length, and the digital camera 100 is moving significantly, so the user is likely to lose sight of the subject. For this reason, it is desirable to perform tracking control process based on the irregularity of the viewpoint trajectory. After the process of S507 is performed here, the process proceeds to S501 in this flowchart.
[0106] In S1104, the system control unit 50 executes the processes of S508 to S513 which do not include the tracking control process of the first embodiment.
[0107] In this embodiment, the tracking control process of S504 to S507 is performed only when the amount of motion of the digital camera 100 is equal to or greater than the threshold THcam and the focal length is equal to or greater than the threshold THd. However, the tracking control process may be performed when at least one of the amount of motion of the digital camera 100 is equal to or greater than the threshold THcam and the focal length is equal to or greater than the threshold THd is satisfied. Also, as in the second embodiment, the system control unit 50 The magnitude of the threshold value THr for determining irregularity in the viewpoint trajectory in S505 may be changed depending on the camera movement and focal length.
[0108] As described above, by switching between execution and non-execution of tracking control processing in accordance with camera movement and lens information, it is possible to appropriately determine the main subject while suppressing the amount of processing.
[0109] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.
[0110] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions. [Explanation of symbols]
[0111] 100: digital camera (tracking device), 50: system control unit, 701: line of sight acquisition unit, 24: image processing unit, 28: display unit, 412: tracking unit
Claims
1. An acquisition means for acquiring a viewpoint position, which is a position where a user is looking at the display means; a tracking means for tracking an object displayed on the display means; a control means for causing the tracking means to perform a tracking process for tracking an object corresponding to the viewpoint position when the degree of irregularity of the change in the viewpoint position is equal to or less than a first threshold, and for causing the tracking means to stop the tracking process when the degree of irregularity of the change in the viewpoint position is greater than the first threshold; A tracking device comprising:
2. and when the degree of irregularity in the change in the viewpoint position is greater than the first threshold, the control means further controls the display means to highlight each object displayed on the display means.
2. The tracking device according to claim 1 .
3. When the degree of irregularity in the change in the viewpoint position is greater than the first threshold value, the control means controls the display means to display a display item indicating each object displayed on the display means based on the viewpoint position.
3. The tracking device according to claim 2.
4. When the degree of irregularity of the change in the viewpoint position is greater than the first threshold value, the control means further controls the device to notify the user that the change in the viewpoint position is irregular.
4. The tracking device according to claim 1, wherein the tracking device is a tracking device for tracking a target object.
5. When the degree of irregularity in the change in the viewpoint position is equal to or less than the first threshold, the control means displays on the display means a display item indicating an object being tracked by the tracking means.
5. The tracking device according to claim 1, wherein the tracking device is a tracking device for tracking a target object.
6. When the degree of irregularity of the change in the display position of the object is equal to or greater than a second threshold, the control means determines whether the degree of irregularity of the change in the display position of the object is less than the second threshold and whether the degree of irregularity of the change in the display position of the object is greater than the second threshold. the first threshold is greater than or equal to a third threshold that is less than the value; 6. The tracking device according to claim 1, wherein the tracking device is a tracking device for tracking a target object.
7. The control means determines an object that the user wishes to track with his / her gaze based on the change in the viewpoint position, using a machine learning device that has performed machine learning for determining an object that the user wishes to track with his / her gaze.
7. The tracking device according to claim 1, wherein the tracking device is a tracking device for tracking a target object.
8. When the tracking means tracks an object, the control means controls the tracking means so as to track the object determined using the machine learning machine.
8. The tracking device according to claim 7.
9. The degree of irregularity of the change in the viewpoint position is a value based on at least one of a frequency, an autocorrelation, a speed, and an acceleration of the change in the viewpoint position.
9. The tracking device according to claim 1, wherein the tracking device is a tracking device for tracking a target object.
10. The degree of irregularity of the change in the viewpoint position is a value based on a cross-correlation between a change in the display position of the object and a change in the viewpoint position.
9. The tracking device according to claim 1, wherein the tracking device is a tracking device for tracking a target object.
11. When the degree of irregularity of the change in the display position of the object is less than a third threshold, the control means causes the tracking means to perform the tracking process regardless of the degree of irregularity of the change in the viewpoint position.
11. The tracking device according to claim 1 ,
12. the control means causes the tracking means to perform the tracking process regardless of the degree of irregularity of the change in the viewpoint position when the movement of the imaging device capturing an image of the object is less than a fourth threshold value; 12. The tracking device according to claim 1, wherein the tracking device is a tracking device for tracking a target object.
13. the control means causes the tracking means to perform the tracking process regardless of the degree of irregularity of the change in the viewpoint position when a focal length of an imaging device that captures an image of the object is less than a fifth threshold value.
13. A tracking device according to claim 1 .
14. A tracking method performed by a tracking device having a tracking means for tracking an object displayed on a display means, comprising: an acquisition step of acquiring a viewpoint position, which is a position where a user is looking at the display means; a control step of causing the tracking means to perform a tracking process for tracking an object corresponding to the viewpoint position when the degree of irregularity of the change in the viewpoint position is equal to or less than a first threshold, and causing the tracking means to stop the tracking process when the degree of irregularity of the change in the viewpoint position is greater than the first threshold; A tracking method comprising:
15. A program for causing a computer to function as each of the means of the tracking device according to any one of claims 1 to 13.
Citation Information
Patent Citations
Photographed object recognizing device and camera
JP1993053043A
Image processing device, imaging device, and control method
JP2018129659A
Information processing apparatus, information processing method, and program
JP2019204503A
Subject recognition device
JP2020106552A
Fixation position recognition device, imaging device, display device, fixation position recognition method, and program
WO2018016209A1
