Subject tracking device, imaging apparatus, subject tracking method, and program

Abstract

To reduce the time until subject tracking resumes when video information acquisition is temporarily halted and then restarted during the execution of field-of-view adjustment for subject tracking based on video information.SOLUTION: A network camera 100 can transition between a first state where it outputs a captured image signal and a second state where it does not output a captured image signal. A camera engine 110 performs imaging of a subject being tracked. A field of view adjustment unit 130 adjusts the field of view of the camera engine 110. A CPU 101 performs subject tracking control based on video information from the camera engine 110. A positioning unit 108 measures the subject's position and distance using radio wave signals. In the second state, the CPU 101 determines the field of view on the basis of positioning information acquired by the positioning unit 108 and controls the field of view adjustment so that the subject is captured within the field of view.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology that achieves both subject tracking control and power saving control of an apparatus. [Background technology]

[0002] An imaging device with an automatic tracking function can identify characteristic image locations extracted from video information and automatically adjust the angle of view so that the object being tracked remains within the angle of view even if it moves.The imaging device performs control to instantly change the angle of view in response to the movement of the object so that the object continues to be captured at the same position within the angle of view.

[0003] On the other hand, there are imaging devices that have a power-saving function that reduces power consumption when video information is temporarily not needed during shooting. The imaging device can transition to a state in which it does not output captured image signals (hereinafter referred to as an "output stop state"), and then transition to a state in which it outputs captured image signals as needed (hereinafter referred to as an "output state").

[0004] In an imaging device with an automatic tracking function and a power saving function, if the automatic angle of view adjustment function is enabled and the device transitions to an output stop state, the device is unable to acquire video information, and the angle of view cannot be adjusted temporarily. If the subject to be identified moves outside the angle of view in this state, the device acquires video information that does not include the image area of ​​the subject when the device transitions to an output state of a captured image signal, and the device is unable to resume automatic angle of view adjustment.

[0005] Therefore, a method can be considered in which the image capture device controls the angle of view within an adjustable range, searches for a specific subject, and re-detects the subject, thereby transitioning to a state where automatic angle of view adjustment is possible again. However, there is a problem in that searching the entire adjustable range of the image capture device in the subject search process takes a very long time. Patent Document 1 discloses a technology in which the angle of view adjustment range used when the image capture device previously output a captured image signal is stored in advance, and when the state transitions from a state where output of the captured image signal is stopped to a state where it is output, the angle of view adjustment range is given priority when searching for the subject. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2021-166368 Summary of the Invention [Problem to be solved by the invention]

[0007] In conventional technology, when an imaging device stops acquiring video information while automatically adjusting the angle of view to track a moving subject, and then starts acquiring video information again, it may take some time for the tracking operation of the subject to resume. The present invention aims to shorten the time until subject tracking operation is resumed when acquisition of video information is temporarily stopped and then resumed while angle of view adjustment for subject tracking based on video information is being performed. [Means for solving the problem]

[0008] An embodiment of the present invention provides a subject tracking device that tracks a subject using video information, and includes: an acquisition means that acquires an image of the subject from an imaging means that can transition between a first state in which an image signal is output and a second state in which an image signal is not output; a positioning means that acquires positioning information of the subject; an adjustment means that adjusts the imaging direction or imaging range of the imaging means; and a control means that controls the adjustment means to adjust the image of the subject to fit within the angle of view of the imaging means, wherein the control means determines the angle of view using the positioning information in the second state, and controls the adjustment means to track the subject. [Effects of the Invention]

[0009] According to the present invention, when the acquisition of image information is temporarily stopped and then restarted while adjusting the angle of view for tracking a subject based on image information, the time until the subject tracking operation is resumed can be shortened. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an overall configuration of a hardware configuration of an imaging apparatus according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating a functional configuration of the imaging apparatus according to the embodiment. [Figure 3] 10A and 10B are diagrams illustrating a subject tracking process based on video information and positioning information. [Figure 4] 10 is a flowchart illustrating the operation of the imaging apparatus according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. While the present invention will be described using a network camera as an example of an imaging device according to the embodiment, the present invention can also be applied to subject tracking devices that use captured images, various imaging devices that have an automatic subject tracking function, and the like.

[0012] [First embodiment] FIG. 1 is a block diagram illustrating an example of the hardware configuration of a network camera 100 according to this embodiment. The components of the network camera 100 are broadly divided into a camera engine 110, a network engine 120, and a field-of-view adjustment unit 130. The camera engine 110 includes an imaging optical system, an imaging element, and a signal processing device (not shown), and outputs a captured image signal in a data format that is easy for the network engine 120 to process. The network engine 120 acquires video data from the camera engine 110, performs image processing and encoding, and outputs the encoded data to an external device (not shown) via a network. The field-of-view adjustment unit 130 adjusts the imaging direction and imaging range of the camera engine 110. For example, the field-of-view adjustment unit 130 changes the physical orientation of the imaging unit (200 in FIG. 2) of the camera engine 110 using a panning mechanism and a tilting mechanism. Regarding the zooming in and out of an image, the field-of-view adjustment unit 130 performs optical adjustment using a zoom mechanism or performs electronic zoom control using image processing.

[0013] The configuration of the network engine 120 will be described with reference to FIG. 1. The central control section of the network engine 120 includes, for example, a CPU (Central Processing Unit) 101. The CPU 101 executes the software program (hereinafter simply referred to as the "program") of the network engine 120 and controls the entire device. A ROM (Read Only Memory) 102 stores the program executed by the CPU 101. The ROM 102 may be, for example, an electrically rewritable flash memory. A RAM (Random Access Memory) 103 is used to store programs, temporary data, and the like when the CPU 101 performs control. In this embodiment, the CPU 101 transfers the program stored in the ROM 102 to the RAM 103 and executes it to perform software processing. The present invention is not limited to this example, and the encoding process may also be implemented in hardware using a configuration equipped with a hardware encoding circuit.

[0014] The network control unit 104 controls the transmission and reception of data to and from external devices on a network (not shown). The camera communication unit 105 communicates with the camera engine 110 regarding control instructions and receives captured video data. The view angle adjustment communication unit 107 communicates with the view angle adjustment unit 130 regarding control instructions.

[0015] The positioning unit 108 measures the position of the object to be positioned using radio wave information and outputs the position information. Specifically, if the object to be positioned carries a small transmitter using an ultra-wideband (UWB) wireless system, a known technique is used to determine the direction and distance using the radio waves received by the device performing the positioning. Another method is to obtain the object's positioning information by detecting electromagnetic waves (such as infrared rays) or sound emitted by the object.

[0016] The bus 109 is electrically connected to the CPU 101, ROM 102, RAM 103, network control unit 104, camera communication unit 105, angle of view adjustment communication unit 107, and positioning unit 108, and each unit can communicate necessary information with each other.

[0017] The functional configuration of the network camera 100 will be described with reference to Fig. 2. Each unit showing the functional configuration is realized by the CPU 101 executing a predetermined program. The imaging unit 200 corresponds to the imaging optical system and imaging element of the camera engine 110 in Fig. 1. The imaging unit 200 performs imaging processing, noise reduction processing, color space conversion processing, etc., and outputs video data to the communication unit 210 and the subject tracking unit 230.

[0018] The communication unit 210 acquires video data from the imaging unit 200 and performs encoding processing to facilitate distribution over the network. The encoded data is distributed by the network control unit 104 in accordance with a predetermined protocol. The communication unit 210 also transmits instructions regarding video acquisition from external devices on the network (not shown) to the video acquisition control unit 220. The video acquisition control unit 220 instructs the imaging unit 200 to start and stop outputting video signals in accordance with the instructions regarding video acquisition received by the communication unit 210.

[0019] Subject tracking unit 230 acquires video data from imaging unit 200, extracts an image of the subject to be tracked (hereinafter also referred to as "tracking target"), and outputs instructions to angle-of-view adjustment unit 240 so that the tracking target can always be captured within the angle of view. Subject tracking unit 230 can also output instructions to angle-of-view adjustment unit 240 after correcting the instructions based on positioning information acquired from positioning unit 250, in addition to the video data from imaging unit 200.

[0020] The angle-of-view adjusting section 240 corresponds to the angle-of-view adjusting section 130 in FIG. 1, and changes the imaging direction of the imaging section 200 and adjusts the enlargement or reduction of the image according to instructions from the subject tracking section 230 and the positioning section 250.

[0021] 1, receives radio signals in a specific frequency band transmitted from a transmitting device (not shown), and performs processing to measure the relative position and distance to the network camera 100. The positioning unit 250 can also acquire position information of the tracking target within the angle of view, correct the instruction content for adjusting the angle of view, and output an instruction to the angle of view adjustment unit 240.

[0022] Next, subject tracking processing based on video information and positioning information will be described with reference to Fig. 3. Fig. 3(A) is a diagram illustrating the relationship between the image area used by subject tracking section 230 to track the subject and the positioning information acquired from positioning section 250. An image within the angle of view corresponding to video information 300 is shown in a rectangular frame, and an image of feature area 302 (see the rectangular dotted line frame) extracted from the image area of ​​subject 301 is shown. For example, subject 301 to be tracked is a person, and feature area 302 is the image area corresponding to the person's head.

[0023] Based on the video information 300 acquired by the imaging unit 200, the subject tracking unit 230 performs image extraction processing of the characteristic region 302 to track the subject 301. For example, assume that the angle of view is determined based on the position and size of the characteristic region 302 in the video information 300. In this case, assume that the positioning position 303 (see FIG. 3A) of the tracking target is acquired from the position information measured by the positioning unit 250 based on the radio wave information. For example, the positioning position 303 is a predetermined position on the chest of the person being tracked. The relative position difference 304 represents the relative position difference between a part of the characteristic region 302 (the upper left corner) and the positioning position 303. It is assumed that there is no significant change in the position where the transmitter is attached on the subject 301. The subject tracking unit 230 can calculate the position of the characteristic region 302 by using the positioning position 303 and the relative position difference 304.

[0024] The processing in this embodiment will be described with reference to the flowchart in Fig. 4. Execution of the processing starts when the network camera 100 is tracking a subject using video information and has stopped acquiring video in response to an instruction from another network device (not shown) (output stopped state). The processing then ends when the network camera 100 receives an instruction from the network device to start acquiring video. The following processing is executed based on a program stored in ROM 102. The CPU 101 executes the processing while reading the program from ROM 102, or executes the processing while reading the program from RAM 103 after transferring the program from ROM 102 to RAM 103 in advance. The CPU 101 starts the iterative processing from S4010 and repeatedly executes a series of processing steps from S4020 to S4060.

[0025] In S4020, CPU 101 performs processing to acquire positioning information via radio waves. Specifically, CPU 101 queries positioning unit 108 via bus 109 to acquire the positioning information. For example, the positioning information corresponds to positioning position 303 in FIG. 3A. Next, in S4030, CPU 101 performs processing to calculate a feature region of the image required for video tracking based on the acquired positioning information. For example, feature region 302 in FIG. 3A is calculated. Next, the process proceeds to S4040.

[0026] In S4040, the CPU 101 determines whether the calculated characteristic region is within the angle of view. In the example of Fig. 3A, the characteristic region 302 is entirely contained within the angle of view. If it is determined that the characteristic region is within the angle of view, the process proceeds to S4060.

[0027] On the other hand, if it is determined in S4040 that the characteristic region does not fall within the angle of view, the process proceeds to S4050. FIG. 3B illustrates an example of a state in which a portion of the characteristic region 312 is outside the angle of view. The characteristic region 312 of the subject is an image region corresponding to the person's head, and the positioning position 313 is a predetermined position on the person's chest. It is assumed that the relative position difference 304 remains unchanged from FIG. 3A. In this case, it is determined in S4040 that the characteristic region 312 calculated in S4030 does not fall within the angle of view, and the process proceeds to S4050.

[0028] In S4050, the CPU 101 controls the angle of view adjustment so that the feature region 312 falls within the angle of view. Specifically, the CPU 101 issues an instruction to the angle of view adjustment communication unit 107 via the bus 109, and the angle of view adjustment communication unit 107 transmits a signal to the angle of view adjustment unit 130 in accordance with the instruction. The angle of view adjustment unit 130 executes processing to change the imaging direction of the imaging unit 200. By changing the physical orientation of the imaging unit 200, it is possible to transition to a state in which the feature region 312 falls within the angle of view. Figure 3(C) illustrates an example of a state in which the feature region 322 falls within the angle of view. The feature region 322 of the subject is an image region corresponding to the person's head, and the positioning position is the same as in Figure 3(B). After S4050, the process proceeds to S4060.

[0029] In S4060, the CPU 101 determines whether to end the iterative processing. If the CPU 101 determines to continue the iterative processing, the CPU 101 proceeds to S4010 and continues the processing. If the CPU 101 determines to end the iterative processing, the CPU 101 ends the above series of processing. By repeating the above process, if the tracking target moves outside the field of view as shown in Figure 3(B), the field of view is automatically adjusted based on the positioning results using radio wave information so that the characteristic area required for tracking falls within the field of view.

[0030] In this embodiment, it is possible to control the tracking target so that its characteristic area is always kept within the angle of view by using positioning information without using video information. This makes it possible to quickly resume the tracking process of the subject when the acquisition process of video information is started. Furthermore, when video information is not being acquired, the angle of view is not adjusted as long as the moving tracking target is within the angle of view. Therefore, it is possible to minimize changes in the physical orientation of the imaging unit when controlling the change of the imaging direction.

[0031] [Second embodiment] A second embodiment of the present invention will be described with reference to FIGS. 3 and 4. In this embodiment, the same matters as in the first embodiment will not be described, and differences will be mainly described. Regarding positioning using radio waves, it is possible to acquire distance information of the tracking target depending on the method. Furthermore, if the subject gets too close to the subject tracking device (image capture device), there is a possibility that the characteristic area required for tracking based on video information including the subject image will not fit within the angle of view. Therefore, in this embodiment, the operation when the subject gets close to the image capture device will be described.

[0032] Differences from the first embodiment will be described with reference to Figure 4. In S4020, the CPU 101 acquires positioning information via radio waves from the positioning unit 108 via the bus 109. Specifically, the positioning information corresponds to the positioning position 333 in Figure 3(D). Compared to the state in Figure 3(C) before movement, the subject has not moved in the up, down, left, or right directions of the angle of view, but the subject appears larger in the image because it is now closer.

[0033] Next, in S4030, the CPU 101 calculates a feature region necessary for tracking the subject based on the acquired positioning information. The positioning method of this embodiment makes it possible to acquire information about the distance from the imaging device to the subject. In addition, by using focal length information and the like possessed by the camera engine 110, it is possible to calculate changes in the size of the subject within the angle of view. In FIG. 3(D), a feature region 332 of the subject is calculated.

[0034] Next, in S4040, the CPU 101 determines whether or not the calculated characteristic region 332 falls within the angle of view. If it is determined in S4040 that the characteristic region 332 does not fall within the angle of view, the process proceeds to S4050.

[0035] In S4050, CPU 101 controls the angle of view adjustment so that the characteristic region of the subject falls within the angle of view. Specifically, CPU 101 outputs a zoom value change instruction to angle of view adjustment communication unit 107 via bus 109. In accordance with the change instruction, angle of view adjustment unit 130 changes the angle of view in the zoom-out direction. Figure 3(E) illustrates an example of a state in which characteristic region 342 of the subject falls within the angle of view.

[0036] In this embodiment, by repeating the above process, it is possible to control the characteristic area of ​​the tracking target to always be kept within the field of view by using positioning information (including distance information) without using video information, thereby achieving the same effect as in the first embodiment.

[0037] [Modification of the above embodiment] In the above embodiment, the imaging direction or imaging range is adjusted (specifically, the camera's imaging orientation and zoom operation are controlled) even when the output of the captured image signal is stopped. If an operating sound is generated during this process, the user may become suspicious. Therefore, in the modified embodiment, when the output of the captured image signal is stopped, the CPU 101 controls the operation speed to reduce the operating sound compared to when controlling the angle of view adjustment in the output state.

[0038] Also, assume that, as a result of positioning, it is determined that the subject to be tracked has moved away from the imaging device. In this case, even though the characteristic region of the subject is within the angle of view, the number of pixels in the characteristic region may be insufficient and fall below the threshold, making it impossible to identify the subject. Therefore, in a modified embodiment, CPU 101 outputs an instruction to angle of view adjustment communication unit 107 via bus 109 to change the angle of view in the zoom-in direction, thereby performing control to ensure the number of pixels in the characteristic region that allows the subject to be identified.

[0039] The imaging device of the above embodiment and modified embodiment has a function of tracking a subject using both image information obtained by imaging and positioning information obtained by radio waves, and performs processing to adjust the angle of view to the angle determined from the positioning information when the output of the captured image signal is stopped. Therefore, the imaging device can capture the subject within the angle of view even when the output of the captured image signal is stopped, and when the imaging device transitions to the output state of the captured image signal, it becomes possible to quickly track the subject based on the image information without performing subject search processing.

[0040] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0041] Embodiments of the present disclosure include the following configurations, methods, and programs. [Configuration 1] An object tracking device that tracks an object using video information, an acquisition means for acquiring an image of the subject from an imaging means capable of transitioning between a first state in which an image signal is output and a second state in which an image signal is not output; a positioning means for acquiring positioning information of the subject; an adjustment means for adjusting the imaging direction or imaging range of the imaging means; a control unit that controls the adjustment unit to fit the image of the subject within the angle of view of the imaging unit, The control means determines the angle of view using the positioning information in the second state, and controls the adjustment means to track the subject. A subject tracking device characterized by: [Configuration 2] the positioning means measures the positioning information using radio waves from the subject; The control means performs a process of calculating a feature region of the image of the subject. 2. The subject tracking device according to configuration 1, [Configuration 3] the positioning means performs measurement using radio waves from the subject to acquire distance information about the subject; The control means performs a process of calculating a feature region of the image of the subject. 2. The subject tracking device according to configuration 1, [Configuration 4] The control means calculates the position of the characteristic area from a relative position difference between the positioning information and the characteristic area, and determines an angle of view including the characteristic area in the second state. 4. The subject tracking device according to configuration 2 or 3, [Configuration 5] The adjustment means changes the imaging direction of the imaging means when the characteristic region is not within the angle of view. 5. The subject tracking device according to configuration 4. [Configuration 6] The adjustment means changes the angle of view of the imaging means in a zoom-out direction when the characteristic region is not contained within the angle of view. 5. The subject tracking device according to configuration 4. [Configuration 7] The control means controls the operation speed of the adjustment means in the second state to be slower than that in the first state. 7. The subject tracking device according to any one of configurations 1 to 6, wherein: [Configuration 8] When the control means determines based on the distance information that the subject has moved away from the imaging means, the control means outputs an instruction to the adjustment means to change the angle of view of the imaging means in a zoom-in direction. 4. The subject tracking device according to configuration 3. [Configuration 9] a subject tracking device according to any one of configurations 1 to 8; The imaging means An imaging device characterized by: [Configuration 10] the positioning means acquires the positioning information or distance information of the subject by radio waves from the subject; The adjustment means adjusts the imaging direction or zoom operation of the imaging means in the first and second states. 10. The imaging device according to configuration 9, [Method 1] A subject tracking method executed by a subject tracking device that tracks a subject using video information, an acquisition step of acquiring an image of the subject from an imaging means capable of transitioning between a first state in which an imaged image signal is output and a second state in which an imaged image signal is not output; a positioning step of acquiring positioning information of the subject; a control step of controlling an adjustment means for adjusting an imaging direction or an imaging range of the imaging means by a control means, thereby controlling an adjustment so that an image of the subject is contained within an angle of view of the imaging means, In the control step, the control means determines the angle of view using the positioning information in the second state, and controls the adjustment means to track the subject. A subject tracking method comprising: [program] A program that causes a computer of a subject tracking device to execute each step described in Method 1. [Explanation of symbols]

[0042] 100 network cameras 110 Camera Engine 120 Network Engine 130 Angle of view adjustment section

Claims

1. An object tracking device that tracks an object using video information, an acquisition means for acquiring an image of the subject from an imaging means capable of transitioning between a first state in which an image signal is output and a second state in which an image signal is not output; a positioning means for acquiring positioning information of the subject; an adjustment means for adjusting the imaging direction or imaging range of the imaging means; a control unit that controls the adjustment unit to fit the image of the subject within the angle of view of the imaging unit, The control means determines the angle of view using the positioning information in the second state, and controls the adjustment means to track the subject. A subject tracking device characterized by:

2. the positioning means measures the positioning information using radio waves from the subject; The control means performs a process of calculating a feature region of the image of the subject.

2. The subject tracking device according to claim 1,

3. the positioning means performs measurement using radio waves from the subject to acquire distance information about the subject; The control means performs a process of calculating a feature region of the image of the subject.

2. The subject tracking device according to claim 1,

4. The control means calculates the position of the characteristic area from a relative position difference between the positioning information and the characteristic area, and determines an angle of view including the characteristic area in the second state.

4. The subject tracking device according to claim 2 or 3.

5. The adjustment means changes the imaging direction of the imaging means when the characteristic region is not within the angle of view.

5. The subject tracking device according to claim 4.

6. The adjustment means changes the angle of view of the imaging means in a zoom-out direction when the characteristic region is not contained within the angle of view.

5. The subject tracking device according to claim 4.

7. The control means controls the operation speed of the adjustment means in the second state to be slower than that in the first state.

2. The subject tracking device according to claim 1,

8. When the control means determines based on the distance information that the subject has moved away from the imaging means, the control means outputs an instruction to the adjustment means to change the angle of view of the imaging means in a zoom-in direction.

4. The subject tracking device according to claim 3.

9. The subject tracking device according to claim 1 ; The imaging means An imaging device characterized by:

10. the positioning means acquires the positioning information or distance information of the subject by radio waves from the subject; The adjustment means adjusts the imaging direction or zoom operation of the imaging means in the first and second states.

10. The imaging device according to claim 9.

11. A subject tracking method executed by a subject tracking device that tracks a subject using video information, an acquisition step of acquiring an image of the subject from an imaging means capable of transitioning between a first state in which an imaged image signal is output and a second state in which an imaged image signal is not output; a positioning step of acquiring positioning information of the subject; a control step of controlling an adjustment means for adjusting an imaging direction or an imaging range of the imaging means by a control means, thereby controlling an adjustment so that an image of the subject is contained within an angle of view of the imaging means, In the control step, the control means determines the angle of view using the positioning information in the second state, and controls the adjustment means to track the subject. A subject tracking method comprising:

12. A program that causes a computer of a subject tracking device to execute each step of the process according to claim 11.

Images