Arithmetic unit and arithmetic method

Abstract

To provide a technique for, when a subject to be tracked moves, more easily bringing the subject closer to a target position and size.SOLUTION: An arithmetic unit calculates a control speed for controlling the posture of an imaging apparatus so as to bring a detected position of a subject detected from an image photographed by the imaging apparatus closer to a target position of the subject in the photographed image. In the calculation, when the difference between the detected position and the target position is equal to or more than a first threshold, the arithmetic unit calculates a first control speed as the control speed for the posture according to the difference, and subsequently when the difference is reduced to a second threshold that is less than the first threshold and smaller than the first threshold, calculates a second control speed faster than the first control speed as the control speed for the posture according to the difference.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a calculation technique for tracking photography. [Background technology]

[0002] Generally, a technology is known for cameras known as PTZ cameras that can adjust pan, tilt, and zoom, in which a subject to be tracked (hereinafter referred to as a "tracked subject") specified by a user is detected from a captured image and tracked and photographed. This tracking technology automatically controls pan, tilt, and zoom so that the tracked subject is kept captured at a target position (hereinafter referred to as a "target position") in the photographic composition. By selecting an appropriate mode from multiple control modes based on the moving speed of the tracked subject, etc., it becomes possible to track and photograph subjects moving at different speeds. Patent Document 1 discloses a method for continuously tracking and photographing subjects moving at a wide range of speeds, from slow to fast, by having at least two control modes and switching the control mode based on the moving speed of the tracked subject. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-68183 Summary of the Invention [Problem to be solved by the invention]

[0004] One issue with tracking technology is that if it tracks even the smallest movements of a subject, the image quality will be poor. For this reason, a control is generally used (hereinafter referred to as dead zone control) that slows down the camera's pan, tilt, and zoom speed as the tracked subject approaches the target position, and stops the pan, tilt, and zoom operations when the tracked subject approaches a certain distance from the target position. This improves image quality when the subject shakes in place or moves slightly.

[0005] However, when the above-mentioned dead zone control is performed, another problem occurs: the pan / tilt / zoom speed is slow or stops near the target position, causing the tracked subject to not reach the target position or to reach it slowly.As a result, if the tracked subject is moving a lot, such as walking, the tracked subject may not reach the target position or may reach it slowly, even if the camera's pan / tilt / zoom speed is increased, resulting in poor image quality.

[0006] In the method of Patent Document 1, the mode cannot be switched when the speed of the tracked subject is constant, so there is a risk that the tracked subject will not reach the target position if the speed of the tracked subject is slow. The present invention provides a technology that makes it easier to bring the tracked subject closer to the target position and size when the tracked subject moves. [Means for solving the problem]

[0007] One aspect of the present invention is characterized in that it includes a calculation means for calculating a control speed for controlling the attitude of the imaging device so that a detected position of a subject detected from an image captured by the imaging device approaches a target position of the subject in the captured image, and when a difference between the detected position and the target position is equal to or greater than a first threshold, the calculation means calculates a first control speed as the control speed of the attitude in accordance with the difference, and when, after that state, the difference becomes equal to or greater than a second threshold that is less than the first threshold and smaller than the first threshold, the calculation means calculates a second control speed that is faster than the first control speed as the control speed of the attitude in accordance with the difference. [Effects of the Invention]

[0008] According to the configuration of the present invention, when a subject to be tracked moves, the subject can be made to approach the target position and size more easily. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration. [Figure 2]2 is a block diagram showing an example of the hardware configuration of each of the camera 100 and the controller 200. [Figure 3] 2 is a block diagram showing an example of the software (computer program) configuration of each of the camera 100 and the controller 200. FIG. [Figure 4] 1A is a flowchart illustrating the operation of the camera 100, and FIG. 1B is a flowchart illustrating the operation of the controller 200. [Figure 5] (a) is a graph showing the relationship between distance difference and angular velocity, and (b) is a diagram showing the correspondence between distance difference and angular velocity. [Figure 6] FIG. 10A is a diagram showing an example of a displayed captured image, and FIG. 10B is a diagram showing a state in which a user is setting a new target position. [Figure 7] 1A is a block diagram showing an example of the hardware configuration of a camera 100, and FIG. 1B is a block diagram showing an example of the software (computer program) configuration of the camera 100. FIG. [Figure 8] 1A is a flowchart illustrating the operation of the camera 100, and FIG. 1B is a flowchart illustrating the operation of the controller 200. [Figure 9] (a) is a graph showing the relationship between size difference and zoom speed, and (b) is a diagram showing the correspondence between size difference and zoom speed. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [First embodiment] First, an example of the configuration of a system according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the system according to this embodiment includes a camera 100 and a controller 200 that is a control device for the camera 100, and the camera 100 and the controller 200 are connected to a network 300. As a result, the system according to this embodiment is configured so that the camera 100 and the controller 200 can communicate data with each other via the network 300. The network 300 includes networks such as a LAN (Local Area Network) and the Internet.

[0012] Next, an example of the hardware configuration of each of the camera 100 and the controller 200 will be described using the block diagram of Fig. 2. Note that the configuration shown in Fig. 2 is merely an example of the hardware configuration of the camera 100 and the controller 200, and can be changed / modified as appropriate.

[0013] First, we will explain an example of the hardware configuration of camera 100. Camera 100 has a mechanism capable of panning and tilting to change its own attitude (image capture direction), and changes the attitude by performing panning and tilting based on the result of subject detection from a captured image.

[0014] The CPU 101 executes various processes using computer programs and data stored in the RAM 102. As a result, the CPU 101 controls the overall operation of the camera 100 and executes or controls various processes that will be described as processes performed by the camera 100.

[0015] The RAM 102 is a high-speed storage device such as a DRAM, and has an area for storing computer programs and data loaded from the storage device 103, and an area for storing captured images output from the image processing unit 106. The RAM 102 also has an area for storing various information received from the controller 200 via the network I / F 105, and a work area used by the CPU 101 and the inference unit 104 when executing various processes. In this way, the RAM 102 can provide various areas as needed.

[0016] The storage device 103 is a non-volatile storage device such as a flash memory, HDD, SSD, or SD card. The storage device 103 stores setting data for the camera 100, computer programs and data related to the startup of the camera 100, and computer programs and data related to the basic operation of the camera 100. The storage device 103 also stores computer programs and data for causing the CPU 101 and the inference unit 104 to execute or control various processes described as processes performed by the camera 100.

[0017] The inference unit 104 performs inference processing to estimate the presence or absence, position, etc. of a subject from a captured image. The inference unit 104 is, for example, a computing device specialized for image processing and inference processing, such as a GPU (Graphics Processing Unit). While a GPU is generally effective for use in inference processing, equivalent functions may also be realized by a reconfigurable logic circuit such as an FPGA (Field Programmable Gate Array). Furthermore, the processing of the inference unit 104 may be performed by the CPU 101.

[0018] The network I / F 105 is an interface for connecting to the network 300, and is responsible for communication with an external device such as the controller 200 via a communication medium such as Ethernet (registered trademark). Note that a separate serial communication I / F may be prepared and used for communication.

[0019] Image processing unit 106 converts the image signal output from image sensor 107 into a captured image, which is data in a predetermined format, compresses the captured image as necessary, and outputs it to RAM 102. Note that image processing unit 106 may perform various processes on the image represented by the image signal acquired from image sensor 107, such as image quality adjustments such as color correction, exposure correction, and sharpness correction, and cropping to cut out only a predetermined area. These processes may also be performed in accordance with instructions received from controller 200 via network I / F 105.

[0020] The image sensor 107 receives light reflected from a subject, converts the brightness and color of the received light into electric charges, and outputs an image signal based on the result of the conversion. The image sensor 107 may be, for example, a photodiode, a CCD (Charge Coupled Device) sensor, or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0021] The drive I / F 108 is an interface for transmitting and receiving instruction signals such as control signals to and from the drive unit 109 .

[0022] The drive unit 109 is a drive mechanism for changing the attitude of the camera 100, and includes a mechanical drive system, a drive motor, etc. The drive unit 109 performs pan and tilt operations for changing the attitude of the camera 100 horizontally and vertically, and zoom operations for optically changing the angle of view, in accordance with instructions received from the CPU 101 via the drive I / F 108.

[0023] The CPU 101 , RAM 102 , storage device 103 , inference unit 104 , network I / F 105 , image processing unit 106 , and drive I / F 108 are all connected to a system bus 110 .

[0024] Next, the controller 200 will be described. The controller 200 can receive captured images transmitted from the camera 100 via the network 300, and can transmit to the camera 100 a target position of a tracking subject based on a user operation on the controller 200. Using this system, a user can specify a target position of a tracking subject using the controller 200, and can use the camera 100 to track and capture the tracking subject so as to move it closer to the selected target position.

[0025] The CPU 201 executes various processes using computer programs and data stored in the RAM 202. As a result, the CPU 201 controls the overall operation of the controller 200 and executes or controls various processes that will be described as processes performed by the controller 200.

[0026] The RAM 202 is a high-speed storage device such as a DRAM. The RAM 202 has an area for storing computer programs and data loaded from the storage device 203, and an area for storing various data received from the camera 100 via the network I / F 205. The RAM 202 also has a work area used by the CPU 201 and the inference unit 204 when executing various processes. In this way, the RAM 202 can provide various areas as needed.

[0027] The storage device 203 is a non-volatile storage device such as a flash memory, HDD, SSD, or SD card. The storage device 203 stores setting data for the controller 200, computer programs and data related to the startup of the controller 200, computer programs and data related to the basic operation of the controller 200, etc. The storage device 203 also stores computer programs and data for causing the CPU 201 and the inference unit 204 to execute or control various processes described as processes performed by the controller 200.

[0028] The inference unit 204 performs inference processing to estimate the presence or absence, position, etc. of a subject from a captured image. The inference unit 204 is, for example, a computing device specialized for image processing and inference processing, such as a GPU (Graphics Processing Unit). While a GPU is generally effective for use in inference processing, equivalent functions may also be realized by a reconfigurable logic circuit such as an FPGA (Field Programmable Gate Array). Furthermore, the processing of the inference unit 204 may be performed by the CPU 201.

[0029] The network I / F 205 is an interface for connecting to the network 300, and is responsible for communication with external devices such as the camera 100 via a communication medium such as Ethernet. For example, communication with the camera 100 includes sending control commands to the camera 100, receiving captured images from the camera 100, and the like.

[0030] Display unit 206 is a display unit having a screen such as a liquid crystal screen or a touch panel screen, and is capable of displaying captured images received from camera 100, a setting screen of controller 200, etc. In this embodiment, a case will be described in which display unit 206 is a display unit having a touch panel screen.

[0031] It should be noted that the controller 200 is not limited to having the display unit 206. For example, the display unit 206 may be omitted from the controller 200, and a display device may be connected to the controller 200 to display the captured image, the setting screen of the controller 200, etc.

[0032] The user input I / F 207 is an interface for receiving operations from the user on the controller 200, and includes, for example, a button, a dial, a joystick, a touch panel, and the like.

[0033] The CPU 201, RAM 202, storage device 203, inference unit 204, network I / F 205, display unit 206, and user input I / F 207 are all connected to a system bus 208. Note that the controller 200 may be a PC (personal computer) having a mouse, keyboard, etc. as the user input I / F 207.

[0034] Next, a block diagram of Fig. 3 shows an example of the configuration of software (computer programs) in each of the camera 100 and the controller 200. Note that Fig. 3 does not show general-purpose software such as an operating system.

[0035] The storage device 103 of the camera 100 stores, as software, an imaging unit 301, an inference unit 302, a drive control unit 303, an arithmetic operation unit 304, and a communication unit 305. The CPU 101 loads these pieces of software from the storage device 103 into the RAM 102 as appropriate and uses them.

[0036] The storage device 203 of the controller 200 stores, as software, a user interface unit 306, an inference unit 307, an arithmetic operation unit 308, and a communication unit 309. The CPU 201 loads these pieces of software from the storage device 203 into the RAM 202 as appropriate and uses them.

[0037] The software configuration shown in Fig. 3 is an example, and for example, one functional unit may be divided into multiple functional units based on their functions, or multiple functional units may be integrated into one functional unit. Also, one or more of the functional units shown in Fig. 3 may be implemented in hardware.

[0038] Next, a description will be given of the operations of the camera 100 and the controller 200 in the system according to this embodiment. First, the operation of the camera 100 will be described with reference to the flowchart in FIG.

[0039] In step S401, the CPU 101 reads out the photographing unit 301 from the storage device 103 to the RAM 102, expands the photographing unit 301, and executes the expanded photographing unit 301. As a result, the CPU 101 acquires a photographed image from the image processing unit 106 and stores the photographed image in the RAM 102.

[0040] In step S402, the CPU 101 reads the communication unit 305 from the storage device 103 to the RAM 102, expands the communication unit 305 in the RAM 102, and executes the expanded communication unit 305. As a result, the CPU 101 transmits the captured image stored in the RAM 102 to the controller 200 via the network I / F 105.

[0041] In step S403, the CPU 101 reads the arithmetic operation unit 304 from the storage device 103 to the RAM 102, expands the arithmetic operation unit 304 in the RAM 102, and executes the expanded arithmetic operation unit 304. As a result, the CPU 101 determines whether or not the "target position of the tracking subject" transmitted from the controller 200 has been received via the network I / F 105.

[0042] If the result of this determination is that the target position has been received, CPU 101 shifts the process to step S404, and if the target position has not been received, CPU 101 shifts the process to step S405.

[0043] In step S404, the CPU 101 executes the arithmetic operation unit 304 expanded in the RAM 102. As a result, the CPU 101 stores the target position received from the controller 200 via the network I / F 105 in the RAM 102.

[0044] In step S405, the CPU 101 reads out the inference unit 302 from the storage device 103 to the RAM 102, expands the inference unit 302 in the RAM 102, and executes the expanded inference unit 302. As a result, the CPU 101 inputs the captured image stored in the RAM 102 in step S401 to the inference unit 104, controls the inference unit 104 to detect the tracking subject from the captured image, and stores the detection result (hereinafter referred to as the detection result) in the RAM 102.

[0045] At this time, the inference unit 104 reads a trained model created using a machine learning technique such as deep learning from the storage device 103 into RAM 102 and expands the trained model in RAM 102. The inference unit 104 then inputs a captured image into the trained model and performs arithmetic processing of the trained model to detect the tracked subject from the captured image, and outputs, as a detection result, position information indicating the position of the tracked subject in the captured image. Note that the CPU 101 may also reduce the size of the captured image, and the inference unit 104 may input the reduced captured image into the trained model. This reduces the processing load of the inference unit 104 and enables faster inference processing.

[0046] Here, the detection result of the tracking subject by the inference unit 104 will be described. When the inference unit 104 inputs a captured image into the trained model and performs arithmetic processing on the trained model, it outputs rectangle information defining a rectangle containing the entire body of the tracking subject (for example, the coordinates of the upper left vertex and the lower right vertex of the rectangle) as position information representing the position of the tracking subject in the captured image. Note that the rectangle information is not limited to the entire body of the tracking subject, but may also be information indicating the position of a part of the tracking subject, for example, the position of the person's head or face if the tracking subject is a person. In this case, it is necessary to change the trained model used to a trained model that provides the desired input and output. Furthermore, the position information of the tracking subject is not limited to the coordinates of the upper left vertex and the lower right vertex of the rectangle containing the entire body of the tracking subject, but may be any information capable of defining the position of the tracking subject in the captured image, such as the center coordinates, width, and height of the rectangle.

[0047] The method of detecting the tracking subject from the captured image by the inference unit 104 is not limited to a specific method. For example, the inference unit 104 may use a template matching method in which a template image of the tracking subject is registered in advance in the storage device 103 or the like, and an area in the captured image that has a high similarity to the template image is detected as the area of the tracking subject.

[0048] In step S406, CPU 101 executes arithmetic operation unit 304 expanded in RAM 102. As a result, CPU 101 calculates the distance (distance difference) between the target position stored in RAM 102 and the subject position derived from the detection results of the tracking subject stored in RAM 102. For example, if the target position is the center position of the area of the tracking subject, CPU 101 calculates the center position of the area of the tracking subject as the subject position from the detection results of the tracking subject stored in RAM 102.

[0049] Here, the distance difference is the distance from the target position to the subject position on the captured image, and is measured in pixels. The CPU 101 stores the calculated distance difference in the RAM 102.

[0050] In step S407, CPU 101 executes arithmetic operation unit 304 expanded in RAM 102. As a result, CPU 101 determines whether or not the distance difference stored in RAM 102 is less than a second threshold value. The second threshold value is a threshold value set in advance as a criterion for determining whether the distance difference from the target position to the subject position has become sufficiently small.

[0051] If the result of this determination is that the distance difference stored in RAM 102 is less than the second threshold, CPU 101 transfers processing to step S410, and if the distance difference stored in RAM 102 is not less than the second threshold, CPU 101 transfers processing to step S408.

[0052] Note that the branching condition for the process in step S407 is not limited to determining a threshold value for the distance difference. CPU 101 stops the tracking operation when the distance difference is sufficiently small, and so the process from step S407 onwards may be branched depending on whether the tracking operation has stopped. For example, whether the tracking operation has stopped can be determined by whether an angle sensor (not shown) that measures the rotation angle of drive unit 109 has stopped changing the angle. If the tracking operation has stopped, CPU 101 proceeds to step S410, and if the tracking operation has not stopped, CPU 101 proceeds to step S408.

[0053] As another example, the processing from step S407 onwards may be branched depending on the time elapsed since switching to speed calculation using (Equation 4), which will be described later. In this case, CPU 101 measures the time elapsed since switching to speed calculation using (Equation 4), and if the elapsed time is equal to or greater than a threshold, CPU 101 proceeds to step S410, and if the elapsed time is less than the threshold, CPU 101 proceeds to step S408.

[0054] In step S408, CPU 101 executes arithmetic operation unit 304 expanded in RAM 102. As a result, CPU 101 determines whether the distance difference stored in RAM 102 is less than a first threshold value that is greater than the second threshold value.

[0055] The first threshold is a threshold that is set in advance as a criterion for determining whether the tracking subject simply shook in place or whether the tracking subject has started to move. That is, in this embodiment, if the difference in distance between the target position and the subject position is less than the first threshold, it is determined that the tracking subject simply shook in place, and if it is not less than the first threshold, it is determined that the tracking subject has moved.

[0056] The first threshold is set by reading out a value stored in advance in RAM 102 or storage device 103, but since the relative distance difference for the optimum threshold changes depending on the photographed size of the tracked subject, the first threshold may be set appropriately depending on the photographed size. This allows adaptation even when the zoom is changed during tracking operation.

[0057] If the result of this determination is that the distance difference stored in RAM 102 is less than the first threshold, CPU 101 transfers processing to step S411, and if the distance difference stored in RAM 102 is not less than the first threshold, CPU 101 transfers processing to step S409.

[0058] In step S409, CPU 101 turns on the value of the threshold excess flag, which is a flag that stores the actual result that the distance difference stored in RAM 102 has exceeded the first threshold, stores the flag in RAM 102, and proceeds to step S412. On the other hand, in step S410, CPU 101 turns off the value of the threshold excess flag, stores the flag in RAM 102, and proceeds to step S412. The initial value of the threshold excess flag is set to "off."

[0059] In step S411, the CPU 101 executes the arithmetic operation unit 304 expanded in the RAM 102. As a result, the CPU 101 determines whether the value of the threshold excess flag stored in the RAM 102 is on.

[0060] If the result of this determination is that the value of the threshold excess flag is on, CPU 101 shifts the process to step S413, and if the value of the threshold excess flag is off, CPU 101 shifts the process to step S412.

[0061] In step S412, CPU 101 executes arithmetic operation unit 304 expanded in RAM 102. As a result, CPU 101 converts the distance difference stored in RAM 102 into an angle difference (pan angle and tilt angle of camera 100). For example, CPU 101 approximately calculates the angle per pixel of the captured image using information on the imaging resolution and imaging angle of view of camera 100, and multiplies this by the distance difference to calculate the angle difference. CPU 101 then calculates angular velocities in the pan and tilt directions according to the calculated angle difference. CPU 101 calculates the first angular velocity ω_1 according to the following (Equation 1), where a is the calculated angle difference, g_1 is the first velocity coefficient, and v_1 is the first offset angular velocity.

[0062] ω_1=axg_1+v_1 … (Equation 1) FIG. 5(a) shows a graph representing the relationship between distance difference and angular velocity. The horizontal axis of the graph in FIG. 5(a) represents distance difference, and the vertical axis represents angular velocity. The straight line calculated using the above-described (Equation 1) is line 504, and line 504 assumes a shape in which the angular velocity increases in proportion to the distance difference. Distance difference 502 represents a first dead zone associated with line 504 in (Equation 1). Here, the dead zone is the range of distance difference in which the angular velocity in the pan direction and the angular velocity in the tilt direction are zero. In other words, line 504 in (Equation 1) outputs a negative angular velocity when distance difference is less than 502, but CPU 101 outputs zero as the angular velocity corresponding to line 504 when the distance difference is less than 502.

[0063] Note that even for the same distance difference, the larger the value of the first speed coefficient g_1, the larger the corresponding angular speed, and the smaller the value of the first offset angular speed v_1, the larger the first dead band. The values of the first speed coefficient g_1 and the first offset angular speed v_1 may be determined experimentally, or may be arbitrarily set by the user by operating the controller 200.

[0064] Then, the CPU 101 generates a control command for the drive unit 109 to rotate the camera 100 in the pan direction and / or tilt direction at a first angular velocity in the pan direction and / or tilt direction, and stores the generated control command in the RAM 102.

[0065] In step S413, CPU 101 executes arithmetic operation unit 304 expanded in RAM 102. As a result, CPU 101 calculates the angle difference (pan angle and tilt angle of camera 100) from the distance difference stored in RAM 102, similar to step S412. CPU 101 then calculates the angular velocity in the pan and tilt directions according to the calculated angle difference. If the calculated angle difference is a, the second velocity coefficient is g_2, and the second offset angular velocity is v_2, CPU 101 calculates the second angular velocity ω_2 according to the following (Equation 2).

[0066] ω_2=axg_2+v_2 … (Equation 2) 5(a), the straight line obtained by the above-described (Equation 2) is line 505, and line 505 has a shape such that the angular velocity increases in proportion to the distance difference, similar to the above-described (Equation 1). Distance difference 501 indicates a second dead zone related to line 505 of (Equation 2). In other words, line 505 of (Equation 2) outputs a negative angular velocity when the distance difference is less than 501, but CPU 101 outputs 0 as the angular velocity corresponding to line 505 when the distance difference is less than 501.

[0067] Note that even for the same distance difference, the larger the value of the second speed coefficient g_2, the larger the corresponding angular velocity, and the smaller the value of the second offset angular velocity v_2, the larger the second dead band. The values of the second speed coefficient g_2 and the second offset angular velocity v_2 may be determined experimentally, or may be arbitrarily set by the user operating the controller 200. However, the first speed coefficient g_1, the second speed coefficient g_2, the first offset angular velocity v_1, and the second offset angular velocity v_2 are determined so that, within a range equal to or greater than the distance difference 501 and less than the distance difference 503, the angular velocity corresponding to the distance difference X within the range on the line 505 of (Equation 2) is larger than the angular velocity corresponding to the distance difference X on the line 504 of (Equation 1). For example, g_2<g_1、v_2> It is set to satisfy v_1.

[0068] Then, the CPU 101 generates a control command for the drive unit 109 to rotate the camera 100 in the pan direction and / or tilt direction at a second angular velocity in the pan direction and / or tilt direction, and stores the generated control command in the RAM 102.

[0069] In view of the above, the correspondence relationship between distance difference and angular velocity is shown in Fig. 5(b). In this embodiment, distance difference 501, at which the distance difference from the target position to the subject position is small enough to stop the tracking operation, is set as the second threshold, and distance difference 503, at which the distance difference from the target position to the subject position is large enough to determine that the subject being tracked has moved, is set as the first threshold.

[0070] 5(b), line 504 is essentially a function that outputs 0 as the first angular velocity when the distance difference is less than distance difference 502, and outputs the first angular velocity calculated according to the above (Equation 1) when the distance difference is equal to or greater than distance difference 502. Line 505 is essentially a function that outputs 0 as the second angular velocity when the distance difference is less than distance difference 501, and outputs the second angular velocity calculated according to the above (Equation 2) when the distance difference is equal to or greater than distance difference 501 and less than distance difference 503.

[0071] As a method for calculating the angular velocity when the distance difference is 503 or more, there is no restriction as to whether to use the above-mentioned (Equation 1) or (Equation 2), but in this embodiment, the angular velocity is calculated using (Equation 1). This is because when the distance difference is 503 or more, the result of (Equation 1) for the angular velocity in the pan direction and tilt direction is greater than the result of (Equation 2), making it easier to continue tracking even a fast-moving tracking subject.

[0072] Therefore, in the example of Figure 5(b), if the distance difference is less than distance difference 501, the angular velocity is 0, and if the distance difference is greater than or equal to distance difference 503, the angular velocity is the first angular velocity calculated according to the above (Equation 1).

[0073] Furthermore, when the distance difference changes from "less than distance difference 501" to "greater than distance difference 501 and less than distance difference 503," the angular velocity becomes the first angular velocity calculated according to the above (Equation 1) (however, when the distance difference is greater than distance difference 501 and less than distance difference 502, the angular velocity becomes 0 due to the dead zone processing).

[0074] Furthermore, when the distance difference changes from "distance difference 503 or more" to "distance difference 501 or more and less than distance difference 503", the angular velocity becomes the second angular velocity calculated according to the above (Equation 2).

[0075] In this embodiment, as shown in FIG. 5B, the distance difference at the intersection of the lines 504 and 505 is set to be the distance difference 503, and when the distance difference exceeds the distance difference 503, the line to be used is switched between the lines 504 and 505. As a result, the angular velocities of the lines 504 and 505 when the distance difference is the distance difference 503 are the same. Therefore, even when the line to be used is switched from the line 504 to the line 505 using the distance difference 503 as the first threshold, it is possible to prevent the output angular velocity from being discontinuous over time. However, the distance difference 503 is not limited to this, and may be a value larger or smaller than that of the intersection. In this case, the line 504 may be gradually switched to the line 505 so that the output angular velocity is not discontinuous over time.

[0076] By performing the processing in steps S407 to S413, for example, when the tracking operation starts, the tracking operation starts slowly at a small angular velocity in the pan / tilt direction calculated by (Equation 1). Then, during tracking, the attitude of camera 100 is controlled so as to reduce the difference in distance, and as the difference in distance increases, the angular velocity in the pan / tilt direction calculated by (Equation 1) becomes large. Then, when the tracking operation begins to stop after the difference in distance exceeds the first threshold, the control method is switched, and the angular velocity in the pan / tilt direction becomes large, as calculated by (Equation 2), so that the difference in distance decreases further. This makes it easier to bring the subject position closer to the target position.

[0077] In this embodiment, the control method is described as a method for calculating the linear angular velocity shown in (Equation 1) and (Equation 2) above, but the calculation method is not limited to this. For example, the angular velocity may be calculated using two quadratic curves that have the same relationship as (Equation 1) and (Equation 2) above. This enables a more gradual start of tracking operation and an operation that makes it easier to continue tracking when the distance difference increases during tracking.

[0078] In step S414, CPU 101 reads out drive control unit 303 from storage device 103 to RAM 102, loads drive control unit 303 in RAM 102, and executes the loaded drive control unit 303. This causes CPU 101 to derive drive parameters for panning and tilting in a desired direction at a desired angular velocity from the control commands stored in RAM 102. Here, the drive parameters refer to parameters for driving and controlling motors (not shown) for the pan direction and tilt direction included in drive unit 109. Next, CPU 101 controls the drive of drive unit 109 via drive I / F 108 based on the derived drive parameters. Drive by drive unit 109 based on the drive parameters causes camera 100 to change the shooting direction (attitude), i.e., perform panning and tilting operations.

[0079] In step S415, CPU 101 executes arithmetic operation unit 304 expanded in RAM 102. As a result, CPU 101 determines whether or not a termination condition for terminating tracking has been satisfied. Various conditions can be applied as the termination condition, and it is not limited to a specific condition. Examples of the termination condition include "a tracking termination instruction has been received from controller 200," "the current date and time has reached a specified date and time," and "a specified time has elapsed since tracking began."

[0080] If the end condition is satisfied as a result of this determination, CPU 101 ends the processing according to the flowchart of Fig. 4(a). On the other hand, if the end condition is not satisfied, CPU 101 advances the processing to step S401.

[0081] Next, the operation of the controller 200 will be described with reference to the flowchart of FIG.

[0082] In step S416, the CPU 201 reads the arithmetic operation unit 308 from the storage device 203 to the RAM 202, loads the calculation operation unit 308 in the RAM 202, and executes the loaded arithmetic operation unit 308. This allows the CPU 201 to determine whether or not the captured image transmitted from the camera 100 has been received via the network I / F 205. Note that the captured image may be received via a video input I / F (not shown) of the controller 200 instead of via the network I / F 205. In that case, the video input I / F may be connected to a video output I / F (not shown) of the camera 100 via a video transmission cable. This makes it possible to adapt to a case where the communication bandwidth of the network 300 is insufficient for sending and receiving the captured image.

[0083] If the result of this determination is that a captured image has been received from camera 100, CPU 201 stores the received captured image in RAM 202 and proceeds to step S417. On the other hand, if a captured image has not been received from camera 100, CPU 201 proceeds to step S416.

[0084] In step S417, the CPU 201 reads the user interface unit 306 from the storage device 203 to the RAM 202, expands the user interface unit 306 in the RAM 202, and executes the expanded user interface unit 306. As a result, the CPU 201 causes the display unit 206 to display the captured image stored in the RAM 202.

[0085] Fig. 6(a) shows an example of a captured image displayed on display unit 206. As shown in Fig. 6(a), a captured image including a tracking subject 602 is displayed on the display screen of display unit 206, and an icon 601 indicating the current target position in capturing the tracking subject 602 is superimposed on the captured image. By looking at the display screen of display unit 206, the user can confirm the image captured by camera 100 and the target position.

[0086] In step S418, the CPU 201 executes the user interface unit 306 deployed in the RAM 202. This causes the CPU 201 to accept a touch operation by the user on the display unit 206 for "setting the target position of the tracking subject." FIG. 6(b) shows how the user sets a new target position in the display example of FIG. 6(a). In FIG. 6(b), the user touches a position different from the target position indicated by icon 601 with their finger, thereby setting that position as the new target position. Icon 603 indicates the target position newly set in response to the user's operation.

[0087] The method for setting the target position is not limited to a specific method. For example, the user may set the target position by operating the user input I / F 207. The CPU 201 then determines whether or not a touch operation for "setting the target position of the tracking subject" has been input.

[0088] If the result of this determination is that a touch operation for "setting the target position of the tracking subject" has been input, CPU 201 stores the position on the captured image where the touch operation has been input as a new target position in RAM 202, and proceeds to step S419. On the other hand, if a touch operation for "setting the target position of the tracking subject" has not been input, CPU 201 proceeds to step S416.

[0089] In this embodiment, the target of the tracking subject is defined as a "position," but the target is not limited to this. For example, the target of the tracking subject may be set as an "area" (target area). When the subject position enters the target area, it is considered to have reached the target position. This is effective because it can suppress excessive tracking operation in response to minute vibrations in the subject position.

[0090] In step S419, the CPU 201 reads the communication unit 309 from the storage device 203 to the RAM 202, expands the communication unit 309 in the RAM 202, and executes the communication unit 309. As a result, the CPU 201 transmits the new target position stored in the RAM 202 to the camera 100 via the network I / F 205.

[0091] In this way, in this embodiment, whether the subject is moving or not is determined depending on the distance difference between the target position and the subject position, and the attitude control method of the camera 100 is switched depending on the result of the determination. This makes it possible to make it easier to bring the subject position closer to the target position when the subject has moved.

[0092] In other words, in this embodiment, an example of an imaging device is described in which a control speed is calculated to control the attitude of the imaging device so that the detected position of a subject detected from an image captured by the imaging device approaches a target position of the subject in the captured image, and when the difference between the detected position and the target position is equal to or greater than a first threshold, a first control speed is calculated as the attitude control speed in accordance with the difference, and when, after that state, the difference becomes less than the first threshold and equal to or greater than a second threshold that is smaller than the first threshold, a second control speed that is faster than the first control speed is calculated as the attitude control speed in accordance with the difference.

[0093] As described below, this embodiment and the second embodiment may be combined. In this case, when the control speed of the attitude of the imaging device is less than a threshold value, if the difference is equal to or greater than a first threshold value, a first control speed may be calculated as the control speed of the attitude in accordance with the difference, and if, after that state, the difference becomes less than the first threshold value and equal to or greater than a second threshold value, a second control speed may be calculated as the control speed of the attitude in accordance with the difference.

[0094] Furthermore, when the control speed of the attitude of the imaging device is equal to or greater than a threshold value, a first control speed may be calculated as the control speed of the attitude according to the difference, or when the difference becomes less than the second threshold value and equal to or greater than the second threshold value after the difference becomes less than the first threshold value and equal to or greater than the second threshold value, the control speed of the attitude may be set to 0.

[0095] In this embodiment, the distance difference at the time of capturing the captured image acquired in step S401 is calculated based on the captured image, but the method of calculating the distance difference is not limited to this. For example, the cumulative distance difference calculated by integrating the distance differences calculated for each captured image up to the current point in time may be used as the distance difference at the current point in time. This balances the movement speed of the tracked subject with the pan / tilt angular velocity of camera 100, so that even if the distance difference between the target position and the subject position in the captured image does not change, it is possible to determine whether the tracked subject has moved from the cumulative distance difference.

[0096] In this embodiment, camera 100 detects the subject and calculates the drive amount for tracking. However, some or all of these processes may be performed by controller 200. In this case, for example, camera 100 transmits a captured image to controller 200. Controller 200 detects the subject to be tracked from the received captured image, calculates drive parameters for tracking the subject as described above, and transmits the drive parameters to camera 100. Camera 100 then tracks and captures the subject to be tracked as described above in accordance with the received drive parameters. In this case, the processing performed by inference unit 104 of camera 100 may be performed by inference unit 204 of controller 200, and the functions of inference unit 302 and arithmetic operation unit 304 of camera 100 may be performed by inference unit 307 and arithmetic operation unit 308 of controller 200, respectively. As a result, the same effect as described above can be obtained even if camera 100 does not have an inference function.

[0097] [Second embodiment] In this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, it will be assumed that the present embodiment is the same as the first embodiment. In this embodiment, a method will be described in which the angular velocity calculation method is switched based on a comparison of the difference between the target size of the tracking subject in the shooting composition (hereinafter referred to as the target size) and the detected size of the tracking subject (hereinafter referred to as the subject size) with a threshold value, thereby making it easier to bring the subject size closer to the target size. Note that the control described in this embodiment and the control described in the first embodiment may be implemented in combination.

[0098] This embodiment also describes an example in which a configuration is newly provided for measuring the motion speed of the driving unit 109 of the camera 100, and the method of calculating the angular velocity is switched depending on the motion speed. Note that switching the method of calculating the angular velocity depending on the motion speed may be performed in combination with the control described in the first embodiment.

[0099] An example of the hardware configuration of the camera 100 according to this embodiment will be described using the block diagram in Fig. 7(a). Note that the configuration shown in Fig. 7(a) is merely an example of the hardware configuration of the camera 100, and can be changed / modified as appropriate.

[0100] The speed sensor 701 is a sensor that measures the operating speed (driving speed) of the driving mechanism of the driving unit 109 of the camera 100. In this embodiment, the speed sensor 701 acquires the driving speed of the driving unit 109 in units of "degrees / second." The speed sensor 701 is connected to the system bus 110.

[0101] Next, an example of the software (computer program) configuration in the camera 100 is shown in the block diagram of FIG. 7(b). Note that general-purpose software such as an operating system is not shown in FIG. 7(b). Note that the software configuration shown in FIG. 7(b) is just an example; for example, one functional unit may be divided into multiple functional units based on their functions, or multiple functional units may be integrated into one functional unit. Furthermore, one or more of the functional units shown in FIG. 7(b) may be implemented in hardware.

[0102] Next, a description will be given of the operations of the camera 100 and the controller 200 in the system according to this embodiment. First, the operation of the camera 100 will be described with reference to the flowchart in FIG.

[0103] Step S801 is the same as step S401 in FIG. 4, and therefore a description thereof will be omitted.

[0104] In step S802, the CPU 101 reads the arithmetic operation unit 304 from the storage device 103 to the RAM 102, loads the calculation operation unit 304 in the RAM 102, and executes the loaded arithmetic operation unit 304. As a result, the CPU 101 determines whether or not the target size of the tracking subject has been received from the controller 200 via the network I / F 105.

[0105] If the result of this determination is that the target size has been received, the CPU 101 proceeds to step S803, and if the target size has not been received, the CPU 101 proceeds to step S804.

[0106] Here, the target size of the tracking subject refers to the size (size) of the tracking subject that is desired to appear in the captured image, and in this embodiment, is expressed as the ratio of the horizontal length of the area of the tracking subject to the horizontal length of the captured image. In other words, it is the ratio of the width of the tracking subject to the width of the captured image. Note that the target size is not limited to the above width ratio, and may also be the ratio of the height of the tracking subject to the height of the captured image. In other words, the method of calculating the size of the tracking subject in the captured image is not limited to a specific method. Furthermore, the target size is defined in advance, and for example, a size ratio is specified for showing the entire body or only the upper body of the tracking subject in the captured image.

[0107] In step S803, the CPU 101 executes the arithmetic operation unit 304 expanded in the RAM 102. As a result, the CPU 101 stores the target size received from the controller 200 via the network I / F 105 in the RAM 102.

[0108] Step S804 is similar to step S405 in FIG. 4, but in this embodiment, size information indicating the horizontal length of the area of the tracking subject in the captured image is output as the detection result.

[0109] In step S805, CPU 101 executes arithmetic operation unit 304 expanded in RAM 102. As a result, CPU 101 calculates the ratio of the "length indicated by the size information" to the horizontal length of the captured image as the subject size. CPU 101 then calculates the difference (size difference) between the subject size and the target size stored in RAM 102, and stores the size difference in RAM 102.

[0110] In step S806, the CPU 101 reads the measurement unit 702 from the storage device 103 to the RAM 102, loads the measurement unit 702 in the RAM 102, and executes the loaded measurement unit 702. As a result, the CPU 101 measures the drive speed of the drive unit 109 and stores the measured drive speed in the RAM 102.

[0111] In step S807, CPU 101 executes arithmetic operation unit 304 expanded in RAM 102. As a result, CPU 101 determines whether or not the size difference stored in RAM 102 is less than a fourth threshold. The fourth threshold is a threshold set in advance as a criterion for determining whether the size difference between the target size and the subject size has become sufficiently small.

[0112] If the size difference is less than the fourth threshold as a result of this determination, CPU 101 proceeds to step S811. On the other hand, if the size difference is not less than the fourth threshold, CPU 101 proceeds to step S808.

[0113] In step S808, the CPU 101 executes the arithmetic operation unit 304 expanded in the RAM 102. As a result, the CPU 101 determines whether the drive speed of the drive unit 109 stored in the RAM 102 is less than the fifth threshold value.

[0114] If the drive speed is less than the fifth threshold as a result of this determination, CPU 101 shifts the process to step S809, and if the drive speed is not less than the fifth threshold, CPU 101 shifts the process to step S813.

[0115] In step S809, CPU 101 executes arithmetic operation unit 304 expanded in RAM 102. As a result, CPU 101 determines whether the size difference stored in RAM 102 is less than a third threshold value that is greater than the fourth threshold value. The third threshold value is a threshold value that is set in advance as a criterion for determining whether the tracking subject simply shook in place or started to move.

[0116] If the result of this determination is that the size difference stored in RAM 102 is less than the third threshold, CPU 101 transfers processing to step S812, and if the size difference stored in RAM 102 is not less than the third threshold, CPU 101 transfers processing to step S810.

[0117] That is, in this embodiment, if the size difference is less than the third threshold, it is determined that the tracking subject simply shook in place, and if the size difference is not less than the third threshold, it is determined that the tracking subject moved.

[0118] The third threshold is set by reading out a value stored in advance in RAM 102 or storage device 103, but since the relative size difference of the optimal threshold changes depending on the photographed size of the tracked subject, the third threshold may be set appropriately depending on the photographed size. This makes it possible to adapt to cases where the zoom is changed during tracking operation.

[0119] In step S810, CPU 101 turns on the value of the threshold excess flag, which is a flag that stores the actual result that the size difference stored in RAM 102 has exceeded the third threshold, stores the flag in RAM 102, and proceeds to step S813. Meanwhile, in step S811, CPU 101 turns off the threshold excess flag, stores the flag in RAM 102, and proceeds to step S813. In this embodiment, the initial value of the threshold excess flag is also set to "off."

[0120] In step S812, the CPU 101 executes the arithmetic operation unit 304 loaded in the RAM 102. As a result, the CPU 101 determines whether the value of the threshold excess flag stored in the RAM 102 is on.

[0121] If the result of this determination is that the value of the threshold excess flag is on, CPU 101 shifts the process to step S814, and if the value of the threshold excess flag is not on, CPU 101 shifts the process to step S813.

[0122] In this case, in addition to determining whether the size difference exceeds the threshold in step S809, the drive speed threshold in step S808 is also determined to be exceeded because this method can be applied to an automatic tracking system that has processing delays. For example, in this system, there is a delay between when the tracking subject stops and when the stop control command is applied, and the camera's drive unit continues to drive by inertia. As a result, if the drive speed exceeds the threshold, the delay is likely to increase the amount of drive by inertia. Therefore, even if the size difference exceeds the third threshold and a calculation method with a high speed is selected, if the drive speed exceeds the fifth threshold, by reselecting a calculation method with a low speed, it is possible to more easily bring the tracking subject closer to the target position, taking into account the amount of drive by inertia due to the delay.

[0123] In step S813, the CPU 101 executes the arithmetic operation unit 304 loaded in the RAM 102. As a result, the CPU 101 converts the size difference stored in the RAM 102 into an angle difference (pan angle and tilt angle of the camera 100). For example, the CPU 101 uses information about the imaging resolution and imaging angle of view of the camera 100 to approximately calculate the angle per 1 percent of the captured image width, and multiplies this by the size difference to calculate the angle difference. The CPU 101 then calculates a zoom speed, which is the speed in the zoom direction according to the calculated angle difference. If the calculated angle difference is a, the first speed coefficient is g_3, and the first offset zoom speed is v_3, the CPU 101 calculates the first zoom speed z_1 according to the following (Equation 3):

[0124] z_1=axg_3+v_3 … (Equation 3) FIG. 9A shows a graph depicting the relationship between size difference and zoom speed. The horizontal axis of the graph in FIG. 9A represents the size difference, and the vertical axis represents the zoom speed. The line obtained using the above-described (Equation 3) is line 904, which assumes a shape such that the zoom speed increases in proportion to the size difference. The size difference 902 represents the first zoom dead zone and the third size difference threshold value associated with line 904 in (Equation 3). In other words, in this embodiment, the size difference 902 serves as both the first zoom dead zone and the third size difference threshold value. Here, the zoom dead zone is the range of size difference within which the zoom speed is set to zero. In other words, when the size difference is less than size difference 902, the CPU 101 outputs 0 as the zoom speed corresponding to line 904.

[0125] Note that even for the same size difference, the larger the value of the first speed coefficient g_3, the higher the corresponding zoom speed, and the smaller the value of the first offset zoom speed v_3, the larger the first dead zone. The values of the first speed coefficient g_3 and the first offset zoom speed v_3 may be determined experimentally, or may be arbitrarily set by the user by operating the controller 200.

[0126] Then, the CPU 101 generates a control command for causing the drive unit 109 to zoom in the zoom direction at the first zoom speed, and stores the generated control command in the RAM 102.

[0127] In step S814, the CPU 101 executes the arithmetic operation unit 304 expanded in the RAM 102. As a result, the CPU 101 calculates the angle difference (pan angle and tilt angle of the camera 100) from the size difference stored in the RAM 102, similar to step S813. The CPU 101 then calculates the zoom speed according to the calculated angle difference. If the calculated angle difference is a, the second speed coefficient is g_4, and the second offset zoom speed is v_4, the CPU 101 calculates the second zoom speed Z_2 according to the following (Equation 4).

[0128] Z_2=axg_4+v_4 … (Equation 4) 5(a), the straight line obtained by the above-mentioned (Equation 4) is line 905, and line 905 has a shape such that the zoom speed increases in proportion to the size difference, similar to the above-mentioned (Equation 3). Size difference 901 indicates a second dead zone related to line 905 of (Equation 4). When the size difference is less than size difference 901, CPU 101 outputs 0 as the zoom speed corresponding to line 905.

[0129] Furthermore, the zoom speed 903 in the graph shown in FIG. 9(a) indicates a fifth threshold value of the speed.

[0130] Note that even for the same size difference, the larger the value of the second speed coefficient g_4, the higher the corresponding zoom speed, and the smaller the value of the second offset zoom speed v_4, the larger the second dead band. The values of the second speed coefficient g_4 and the second offset zoom speed v_4 may be determined experimentally, or may be arbitrarily set by the user operating the controller 200. However, the first speed coefficient g_3, the second speed coefficient g_4, the first offset zoom speed v_3, and the second offset zoom speed v_4 are determined so that, within a range equal to or greater than the size difference 901 and less than the size difference 902, the zoom speed corresponding to the size difference X within that range on the straight line 905 of (Equation 4) is greater than the zoom speed corresponding to the size difference X on the straight line 904 of (Equation 3). For example, g_4<g_3、v_4> It is set to satisfy v_3.

[0131] Then, the CPU 101 generates a control command for causing the drive unit 109 to zoom in the zoom direction at the second zoom speed, and stores the generated control command in the RAM 102.

[0132] In view of the above, the correspondence relationship between size difference and zoom speed is shown in Fig. 9(b) In this embodiment, the fourth threshold value is set to a size difference 901 where the size difference between the target size and the subject size is small enough to stop the tracking operation, and the third threshold value is set to a size difference 902 where the size difference between the target size and the subject size is large enough to determine that the subject being tracked has moved.

[0133] 9(b), the straight line 904 is essentially a function that outputs 0 as the first zoom speed when the size difference is less than the size difference 902, and outputs the first zoom speed calculated according to the above (Equation 3) when the size difference is equal to or greater than the size difference 902. The straight line 905 is essentially a function that outputs 0 as the second zoom speed when the size difference is less than the size difference 901, and outputs the second zoom speed calculated according to the above (Equation 4) when the size difference is equal to or greater than the size difference 901 but less than the size difference 902.

[0134] Note that, although there is no limitation as to whether to use the above-mentioned (Equation 3) or (Equation 4) as a method for calculating the zoom speed when the size difference is 902 or more, in this embodiment the zoom speed is calculated using (Equation 3). This is because when the size difference is 902 or more, the result of (Equation 3) is greater than the result of (Equation 4) in terms of the zoom speed, making it easier to continue tracking even a fast-moving tracking subject.

[0135] 9(b), if the size difference is less than size difference 901, the zoom speed is 0, and if the size difference is equal to or greater than size difference 902, the zoom speed is the first zoom speed calculated according to the above (Equation 3). Also, if the size difference is "equal to or greater than size difference 901 and less than size difference 902," the method for calculating the zoom speed differs depending on whether the drive speed is equal to or greater than the fifth threshold value or less than the fifth threshold value.

[0136] If the drive speed is equal to or greater than the fifth threshold, the zoom speed is the first zoom speed calculated according to Equation 3 above (but is 0 due to the dead zone processing). On the other hand, if the drive speed is less than the fifth threshold, the zoom speed is calculated as follows:

[0137] When the size difference changes from "less than size difference 901" to "greater than size difference 901 and less than size difference 902," the zoom speed becomes the first zoom speed calculated according to the above (Equation 3) (however, it becomes 0 due to the dead zone processing).

[0138] Furthermore, when the size difference changes from "size difference 902 or more" to "size difference 901 or more and less than size difference 902," the zoom speed becomes the second zoom speed calculated according to the above (Equation 4).

[0139] By performing the processes in steps S807 to S814, for example, when the tracking operation starts, the tracking operation starts slowly at a low zoom speed calculated by (Equation 3). In other words, tracking starts when the size difference exceeds the first dead zone. Next, during tracking, the zoom control of camera 100 is performed to reduce the size difference, and as the size difference increases, the zoom speed calculated by (Equation 3) becomes high. Then, when the tracking operation starts to stop after the size difference exceeds the third threshold, the control method is switched and the zoom speed calculated by (Equation 4) becomes high so that the size difference decreases further. This makes it easier to bring the subject size closer to the target size.

[0140] In this embodiment, the method for calculating the control method has been described as a method for calculating the linear zoom speed shown in (Equation 3) and (Equation 4) above, but the calculation method is not limited to this. For example, the zoom speed may be calculated using two quadratic function curves that have the same relationship as (Equation 3) and (Equation 4) above. This enables a more gradual start of tracking operation and easier continued tracking when the size difference increases during tracking.

[0141] In step S815, the CPU 101 reads out the drive control unit 303 from the storage device 103 to the RAM 102, loads the drive control unit 303 in the RAM 102, and executes the loaded drive control unit 303. As a result, the CPU 101 derives drive parameters for zooming in the zoom direction at a desired zoom speed from the control command stored in the RAM 102. Here, the drive parameters refer to parameters for driving and controlling a zoom direction motor (not shown) included in the drive unit 109. Next, the CPU 101 drives and controls the drive unit 109 via the drive I / F 108 based on the derived drive parameters. The drive unit 109 drives based on the drive parameters, causing the camera 100 to change the shooting magnification (zoom change), i.e., to perform a zoom operation.

[0142] In step S816, CPU 101 determines whether or not the termination condition for terminating tracking is satisfied, similarly to step S415 above. If the termination condition is satisfied as a result of this determination, CPU 101 terminates the processing according to the flowchart in Fig. 8(a). On the other hand, if the termination condition is not satisfied, CPU 101 proceeds to step S801.

[0143] Next, the operation of the controller 200 will be described with reference to the flowchart of FIG. 8(b).

[0144] In step S817, CPU 201 reads user interface unit 306 from storage device 203 to RAM 202, expands user interface unit 306 in RAM 202, and executes expanded user interface unit 306. As a result, CPU 201 causes display unit 206 to display the captured image received from camera 100 and stored in RAM 202. Then, CPU 201 executes user interface unit 306 expanded in RAM 202. As a result, CPU 201 accepts a touch operation on display unit 206 by the user for "setting the target size of the tracking subject."

[0145] For example, the user may touch a preset target size selection item (displayed on the display unit 209) such as full body, upper body, or bust shot with his / her finger to set the target size corresponding to the selected item. Alternatively, the user may input a numerical value as the target size.

[0146] The method for setting the target size is not limited to a specific method. For example, the user may set the target size by operating the user input I / F 207. The CPU 201 then determines whether or not a touch operation for "setting the target size of the tracking subject" has been input.

[0147] If it is determined that a touch operation for "setting the target size of the tracking subject" has been input, CPU 201 stores the target size set by the touch operation in RAM 202 and proceeds to step S818. On the other hand, if a touch operation for "setting the target size of the tracking subject" has not been input, CPU 201 proceeds to step S817.

[0148] In step S818, the CPU 201 reads the communication unit 309 from the storage device 203 to the RAM 202, expands the communication unit 309 in the RAM 202, and executes the communication unit 309. As a result, the CPU 201 transmits the new target size stored in the RAM 202 to the camera 100 via the network I / F 205.

[0149] In this way, by determining whether the tracking subject has moved or not depending on the size difference between the target size and the subject size and switching the control operation, it becomes possible to make it easier to bring the subject size closer to the target size when the tracking subject has moved.

[0150] In other words, in this embodiment, an example of an imaging device has been described in which a control speed is calculated to control the zoom of the imaging device so that the detected size of a subject detected from an image captured by the imaging device approaches the target size of the subject in the captured image, and when the difference between the detected size and the target size is equal to or greater than a first threshold, a first control speed is calculated as the zoom control speed in accordance with the difference, and when, after that state, the difference becomes less than the first threshold and equal to or greater than a second threshold that is smaller than the first threshold, a second control speed that is faster than the first control speed is calculated as the zoom control speed in accordance with the difference.

[0151] In addition, after the difference is less than the second threshold, if the difference becomes less than the first threshold and greater than or equal to the second threshold, the first control speed may be calculated as the zoom control speed in accordance with the difference.

[0152] Also in this embodiment, part or all of the calculation of the drive amount for detecting and tracking the subject may be performed by controller 200. In this case, camera 100 first transmits a captured image to controller 200. Controller 200 detects the subject to be tracked from the received captured image, calculates drive parameters for tracking the subject to be tracked as described above, and transmits the drive parameters to camera 100. Camera 100 then tracks and captures the subject to be tracked as described above in accordance with the received drive parameters.

[0153] As described above, the processing for calculating the control speed (angular speed and zoom speed) described in the first and second embodiments may be performed by camera 100, by controller 200, or by a device separate from these devices. In other words, the processing device that executes the processing for calculating the control speed (angular speed and zoom speed) may be incorporated into camera 100, may be incorporated into controller 200, or may exist as a device separate from these devices. Furthermore, such a processing device may be configured with hardware, may be configured with software, or may be configured with a combination of hardware and software.

[0154] Also, for example, the zoom of camera 100 may be stopped by determining a threshold value for the distance difference, and the pan / tilt of camera 100 may be stopped by determining a threshold value for the size difference. This allows the pan / tilt / zoom of camera 100 to be moved simultaneously, thereby improving the image quality.

[0155] The numerical values, processing timing, processing order, processing subject, data (information) configuration / acquisition method / sending destination / sending source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0156] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0157] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0158] The invention of this specification includes the following arithmetic device, arithmetic method, and computer program. (Item 1) a calculation means for calculating a control speed for controlling the attitude of the imaging device so that a detected position of a subject detected from an image captured by the imaging device approaches a target position of the subject in the captured image; The calculation means When the difference between the detected position and the target position is equal to or greater than a first threshold, a first control speed is calculated as a control speed for the attitude in accordance with the difference, and when, after this state, the difference becomes less than the first threshold and equal to or greater than a second threshold that is smaller than the first threshold, a second control speed that is faster than the first control speed is calculated as a control speed for the attitude in accordance with the difference. A computing device characterized by: (Item 2) 2. The computing device according to item 1, wherein the calculation means sets the attitude control speed to 0 if the difference is less than the second threshold value. (Item 3) 3. The computing device according to item 1 or 2, wherein when the difference becomes less than the first threshold and equal to or greater than the second threshold after the difference is less than the second threshold, the computing means calculates the first control speed as the control speed of the attitude in accordance with the difference. (Item 4) moreover 4. The computing device according to any one of items 1 to 3, further comprising a control means for controlling the attitude in accordance with the control speed calculated by the calculation means. (Item 5) moreover, 4. The arithmetic device according to any one of items 1 to 3, further comprising a transmitting unit that transmits the control speed calculated by the calculating unit to the imaging device. (Item 6) The calculation means calculates the first control speed as the control speed of the attitude of the imaging device in accordance with the difference when the control speed of the attitude of the imaging device is less than a threshold and the difference is equal to or greater than the first threshold, and when the difference becomes less than the first threshold and equal to or greater than the second threshold after that state, calculates the second control speed as the control speed of the attitude in accordance with the difference. (Item 7) 2. The computing device according to item 1, wherein the calculation means calculates the first control speed as the control speed of the attitude of the imaging device in accordance with the difference when the control speed of the attitude of the imaging device is equal to or greater than a threshold value. (Item 8) 2. The computing device according to item 1, wherein the calculation means sets the attitude control speed to 0 when the difference becomes less than the first threshold and equal to or greater than the second threshold after the difference is less than the second threshold. (Item 9) a calculation means for calculating a control speed for controlling the zoom of the imaging device so that a detected size of a subject detected from an image captured by the imaging device approaches a target size of the subject in the captured image; The calculation means When the difference between the detected size and the target size is equal to or greater than a first threshold, a first control speed is calculated as the control speed of the zoom in accordance with the difference, and when, after this state, the difference becomes less than the first threshold and equal to or greater than a second threshold that is smaller than the first threshold, a second control speed that is faster than the first control speed is calculated as the control speed of the zoom in accordance with the difference. A computing device characterized by: (Item 10) Item 10. The arithmetic device according to item 9, wherein when the zoom control speed of the imaging device is less than a threshold value, and the difference is equal to or greater than the first threshold value, the calculation means calculates the first control speed as the zoom control speed in accordance with the difference, and when, after that state, the difference becomes less than the first threshold value and equal to or greater than the second threshold value, the calculation means calculates the second control speed as the zoom control speed in accordance with the difference. (Item 11) 11. The computing device according to item 9 or 10, wherein the calculation means calculates the first control speed as the zoom control speed in accordance with the difference when the zoom control speed of the imaging device is equal to or greater than a threshold. (Item 12) 12. The arithmetic device according to any one of items 9 to 11, wherein the calculation means sets the zoom control speed to 0 if the difference is less than the second threshold value. (Item 13) 13. The computing device according to any one of items 9 to 12, wherein the calculation means sets the zoom control speed to 0 when the difference becomes less than the first threshold and equal to or greater than the second threshold after the difference is less than the second threshold. (Item 14) moreover 14. The computing device according to any one of items 9 to 13, further comprising a control means for controlling the zoom in accordance with the control speed calculated by the calculation means. (Item 15) moreover, 14. The arithmetic device according to any one of items 9 to 13, further comprising a transmitting means for transmitting the control speed calculated by the calculating means to the imaging device. (Item 16) Item 10. The computing device according to item 9, wherein when the difference becomes less than the first threshold and equal to or greater than the second threshold after the difference is less than the second threshold, the computing means calculates the first control speed as the control speed of the zoom in accordance with the difference. (Item 17) A calculation method performed by a calculation device, a calculation step in which a calculation means of the arithmetic unit calculates a control velocity for controlling the attitude of the imaging device so that a detected position of a subject detected from an image captured by the imaging device approaches a target position of the subject in the captured image, In the calculation step, When the difference between the detected position and the target position is equal to or greater than a first threshold, a first control speed is calculated as a control speed for the attitude in accordance with the difference, and when, after this state, the difference becomes less than the first threshold and equal to or greater than a second threshold that is smaller than the first threshold, a second control speed that is faster than the first control speed is calculated as a control speed for the attitude in accordance with the difference. A calculation method characterized by: (Item 18) A calculation method performed by a calculation device, a calculation step in which a calculation means of the arithmetic unit calculates a control speed for controlling the zoom of the imaging device so that a detected size of a subject detected from an image captured by the imaging device approaches a target size of the subject in the captured image, In the calculation step, When the difference between the detected size and the target size is equal to or greater than a first threshold, a first control speed is calculated as the control speed of the zoom in accordance with the difference, and when, after this state, the difference becomes less than the first threshold and equal to or greater than a second threshold that is smaller than the first threshold, a second control speed that is faster than the first control speed is calculated as the control speed of the zoom in accordance with the difference. A calculation method characterized by: (Item 19) A computer program for causing a computer to function as each means of the arithmetic device according to any one of items 1 to 16.

[0159] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0160] 100: Camera 200: Controller 300: Network

Claims

1. a calculation means for calculating a control speed for controlling the attitude of the imaging device so that a detected position of a subject detected from an image captured by the imaging device approaches a target position of the subject in the captured image; The calculation means When the difference between the detected position and the target position is equal to or greater than a first threshold, a first control speed is calculated as a control speed for the attitude in accordance with the difference, and when, after this state, the difference becomes less than the first threshold and equal to or greater than a second threshold that is smaller than the first threshold, a second control speed that is faster than the first control speed is calculated as a control speed for the attitude in accordance with the difference. A computing device characterized by:

2. 2. The computing device according to claim 1, wherein the calculation means sets the attitude control speed to 0 if the difference is less than the second threshold value.

3. 2. The computing device according to claim 1, wherein, when the difference becomes less than the first threshold and equal to or greater than the second threshold after the difference is less than the second threshold, the calculation means calculates the first control speed as the attitude control speed in accordance with the difference.

4. moreover 2. The computing device according to claim 1, further comprising a control means for controlling the attitude in accordance with the control speed calculated by the calculation means.

5. moreover, 2. The computing device according to claim 1, further comprising a transmitting means for transmitting the control speed calculated by the calculating means to the imaging device.

6. 2. The computing device according to claim 1, wherein when the control speed of the attitude of the imaging device is less than a threshold value and the difference is equal to or greater than the first threshold value, the calculation means calculates the first control speed as the control speed of the attitude according to the difference, and when, after that state, the difference becomes less than the first threshold value and equal to or greater than the second threshold value, the calculation means calculates the second control speed as the control speed of the attitude according to the difference.

7. 2. The computing device according to claim 1, wherein the calculation means calculates the first control speed as the attitude control speed of the imaging device in accordance with the difference when the attitude control speed of the imaging device is equal to or greater than a threshold value.

8. 2. The computing device according to claim 1, wherein the calculation means sets the attitude control speed to 0 when the difference becomes less than the first threshold and equal to or greater than the second threshold after the difference becomes less than the second threshold after the difference is less than the first threshold.

9. a calculation means for calculating a control speed for controlling the zoom of the imaging device so that a detected size of a subject detected from an image captured by the imaging device approaches a target size of the subject in the captured image; The calculation means When the difference between the detected size and the target size is equal to or greater than a first threshold, a first control speed is calculated as the control speed of the zoom in accordance with the difference, and when, after this state, the difference becomes less than the first threshold and equal to or greater than a second threshold that is smaller than the first threshold, a second control speed that is faster than the first control speed is calculated as the control speed of the zoom in accordance with the difference. A computing device characterized by:

10. 10. The computing device according to claim 9, wherein when the zoom control speed of the imaging device is less than a threshold value and the difference is equal to or greater than the first threshold value, the calculation means calculates the first control speed as the zoom control speed in accordance with the difference, and when, after that state, the difference becomes less than the first threshold value and equal to or greater than the second threshold value, the calculation means calculates the second control speed as the zoom control speed in accordance with the difference.

11. 10. The arithmetic device according to claim 9, wherein the calculation means calculates the first control speed as the zoom control speed in accordance with the difference when the zoom control speed of the imaging device is equal to or greater than a threshold value.

12. 10. The computing device according to claim 9, wherein the calculation means sets the zoom control speed to 0 if the difference is less than the second threshold value.

13. The computing device according to claim 9, wherein the calculation means sets the zoom control speed to 0 when the difference becomes less than the first threshold and equal to or greater than the second threshold after the difference is less than the second threshold.

14. moreover 10. The computing device according to claim 9, further comprising a control means for controlling the zoom in accordance with the control speed calculated by the calculation means.

15. moreover, 10. The computing device according to claim 9, further comprising a transmitting unit that transmits the control speed calculated by the calculating unit to the imaging device.

16. The computing device according to claim 9, wherein when the difference becomes less than the first threshold and greater than or equal to the second threshold after the difference is less than the second threshold, the calculation means calculates the first control speed as the control speed of the zoom in accordance with the difference.

17. A calculation method performed by a calculation device, a calculation step in which a calculation means of the arithmetic unit calculates a control velocity for controlling the attitude of the imaging device so that a detected position of a subject detected from an image captured by the imaging device approaches a target position of the subject in the captured image, In the calculation step, When the difference between the detected position and the target position is equal to or greater than a first threshold, a first control speed is calculated as a control speed for the attitude in accordance with the difference, and when, after this state, the difference becomes less than the first threshold and equal to or greater than a second threshold that is smaller than the first threshold, a second control speed that is faster than the first control speed is calculated as a control speed for the attitude in accordance with the difference. A calculation method characterized by:

18. A calculation method performed by a calculation device, a calculation step in which a calculation means of the arithmetic unit calculates a control speed for controlling the zoom of the imaging device so that a detected size of a subject detected from an image captured by the imaging device approaches a target size of the subject in the captured image, In the calculation step, When the difference between the detected size and the target size is equal to or greater than a first threshold, a first control speed is calculated as the control speed of the zoom in accordance with the difference, and when, after this state, the difference becomes less than the first threshold and equal to or greater than a second threshold that is smaller than the first threshold, a second control speed that is faster than the first control speed is calculated as the control speed of the zoom in accordance with the difference. A calculation method characterized by:

19. A computer program for causing a computer to function as each of the means of the arithmetic device according to any one of claims 1 to 16.

Images