Arithmetic device, arithmetic method, computer readable storage medium, and computer program product

Abstract

The invention provides an arithmetic device, an arithmetic method, a computer readable storage medium, and a computer program product. At least one apparatus calculates a control speed for controlling a posture of an imaging apparatus such that a detection position of a subject detected in an image captured by the imaging apparatus becomes closer to a target position of the subject in the captured image. In a state in which a difference between the detected position and the target position is not less than a first threshold value, the device calculates a first control speed as a control speed of the posture according to the difference, and in a case after the state that the difference becomes less than the first threshold value and not less than a second threshold value smaller than the first threshold value, the device calculates a second control speed as a control speed of the posture. The device calculates a second control speed higher than the first control speed as a control speed of the posture based on the difference.

Description

Technical Field

[0001] One or more features of the present disclosure generally relate to computing devices, computing methods, and computer-readable storage media, and more particularly to one or more embodiments of computing techniques for tracking and capturing images. Background Art

[0002] In cameras called PTZ cameras that are capable of adjusting pan, pitch, and zoom, there is generally known a technology that detects a subject to be tracked (hereinafter referred to as a tracking subject) specified by a user from a captured image, tracks the subject, and captures an image of the subject. According to this tracking technology, pan, pitch, and zoom are automatically controlled to keep capturing the tracked subject at a target position in the camera composition. At this time, by selecting an appropriate mode from a plurality of control modes based on the moving speed of the tracked subject, etc., it is possible to track subjects with different speeds and capture images of the subjects. Japanese Patent Laid-Open No. 2019-68183 discloses a method as follows: by having at least two control modes and switching between control modes based on the moving speed of the tracked subject, it is possible to keep tracking even a subject with a wide speed range from low speed to high speed and capture an image of the subject.

[0003] Tracking technology can lead to poor video quality if the subject's movements are small. Therefore, as the subject moves closer to the target location, the camera's pan, tilt, and zoom speeds are typically reduced. Furthermore, when the subject moves within a predetermined distance from the target location, control is performed to stop the pan, tilt, and zoom operations (hereinafter referred to as dead zone control). This improves quality when the subject is oscillating in place or moving slightly.

[0004] However, if dead-zone control is implemented, the pan, tilt, and zoom speeds are slow or stopped near the target position, causing the tracked subject to not reach the target position or to arrive at the target position late. Consequently, if the tracked subject walks to increase the amount of movement, even if the pan, tilt, and zoom speeds of the camera are increased, the tracked subject may not reach the target position or arrive at the target position late, deteriorating video quality.

[0005] In the method disclosed in Japanese Patent Laid-Open No. 2019-68183, since the mode cannot be switched when the speed of tracking the subject is constant, the tracking subject may not reach the target position when the speed of tracking the subject is low. Summary of the Invention

[0006] One or more aspects of the embodiments of the present disclosure provide a technique that makes it easier to bring a subject to be tracked closer to a target position or size if the subject moves.

[0007] According to one or more aspects that can be used in one or more embodiments of the present disclosure, a computing device is provided, which may include: a computing unit configured to or operated to calculate a control speed for controlling a posture of a camera device so that a detection position of a subject detected from an image captured by the camera device becomes closer to a target position of the subject in the captured image, wherein, in a state where a difference between the detection position and the target position is not less than a first threshold, the computing unit calculates a first control speed as the control speed of the posture based on the difference, and in a state where the difference becomes less than the first threshold and not less than a second threshold that is smaller than the first threshold after the state, the computing unit calculates a second control speed higher than the first control speed as the control speed of the posture based on the difference.

[0008] According to one or more additional aspects that can be used in one or more embodiments of the present disclosure, a computing device is provided, which may include: a computing unit configured to or operated to calculate a control speed for controlling the zoom of a camera device so that the detected size of a subject detected from an image captured by the camera device becomes closer to a target size of the subject in the captured image, wherein, in a state where the difference between the detected size and the target size is not less than a first threshold, the computing unit calculates a first control speed as the control speed of the zoom based on the difference, and in a case where after the state the difference becomes less than the first threshold and not less than a second threshold that is smaller than the first threshold, the computing unit calculates a second control speed higher than the first control speed as the control speed of the zoom based on the difference.

[0009] According to one or more aspects that can be used in one or more embodiments of the present disclosure, there is provided a computing method, which is executed by a computing device, and the method may include: calculating a control speed for controlling the posture of a camera device so that the detection position of a subject detected from an image captured by the camera device becomes closer to the target position of the subject in the captured image, wherein, in the calculation, in a state where the difference between the detection position and the target position is not less than a first threshold, a first control speed is calculated as the control speed of the posture based on the difference, and in a state where the difference becomes less than the first threshold and not less than a second threshold smaller than the first threshold after the state, a second control speed higher than the first control speed is calculated as the control speed of the posture based on the difference.

[0010] According to one or more additional aspects that can be used in one or more embodiments of the present disclosure, an operation method is provided, which is executed by an operation device, and the method may include: calculating a control speed for controlling the zoom of a camera device so that the detection size of a subject detected from an image captured by the camera device becomes closer to a target size of the subject in the captured image, wherein, in the calculation, in a state where the difference between the detection size and the target size is not less than a first threshold, a first control speed is calculated as the control speed of the zoom based on the difference, and in a state where the difference becomes less than the first threshold and not less than a second threshold which is smaller than the first threshold after the state, a second control speed higher than the first control speed is calculated as the control speed of the zoom based on the difference.

[0011] According to one or more aspects that can be used in one or more embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program for causing a computer to perform the following method, which may include: calculating a control speed for controlling the posture of a camera device so that the detection position of a subject detected from an image captured by the camera device becomes closer to the target position of the subject in the captured image, wherein, in the calculation, in a state where the difference between the detection position and the target position is not less than a first threshold, a first control speed is calculated as the control speed of the posture based on the difference, and in a state where the difference becomes less than the first threshold and not less than a second threshold smaller than the first threshold after the state, a second control speed higher than the first control speed is calculated as the control speed of the posture based on the difference.

[0012] According to one or more additional aspects that can be used in one or more embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program for causing a computer to perform the following method, wherein the method includes: calculating a control speed for controlling the zoom of a camera device so that the detection size of a subject detected from an image captured by the camera device becomes closer to a target size of the subject in the captured image, wherein, in the calculation, in a state where the difference between the detection size and the target size is not less than a first threshold, a first control speed is calculated as the control speed of the zoom based on the difference, and after the state, when the difference becomes less than the first threshold and not less than a second threshold that is smaller than the first threshold, a second control speed higher than the first control speed is calculated as the control speed of the zoom based on the difference.

[0013] According to one or more aspects that can be used in one or more embodiments of the present disclosure, a computer program product is provided, which stores a computer program for causing a computer to perform the following method, the method comprising: calculating a control speed for controlling the posture of a camera device so that a detection position of a subject detected from an image captured by the camera device becomes closer to a target position of the subject in the captured image, wherein, in the calculation, in a state where a difference between the detection position and the target position is not less than a first threshold, a first control speed is calculated as the control speed of the posture based on the difference, and in a state where the difference becomes less than the first threshold and not less than a second threshold which is smaller than the first threshold after the state, a second control speed higher than the first control speed is calculated as the control speed of the posture based on the difference.

[0014] According to one or more additional aspects that can be used in one or more embodiments of the present disclosure, a computer program product is provided, which stores a computer program for causing a computer to perform the following method, the method comprising: calculating a control speed for controlling the zoom of a camera device so that the detection size of a subject detected from an image captured by the camera device becomes closer to a target size of the subject in the captured image, wherein, in the calculation, in a state where the difference between the detection size and the target size is not less than a first threshold, a first control speed is calculated as the control speed of the zoom based on the difference, and in a state where the difference becomes less than the first threshold and not less than a second threshold which is smaller than the first threshold after the state, a second control speed higher than the first control speed is calculated as the control speed of the zoom based on the difference.

[0015] According to other aspects of the present disclosure, one or more additional computing devices, one or more additional computing methods, one or more additional storage media, and one or more computer program products are discussed herein. Other features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a diagram showing an example of the structure of a system according to one or more embodiments of the present disclosure;

[0017] Figure 2 is a block diagram illustrating an example of a hardware structure that may be used for the camera 100 and the controller 200 according to one or more embodiments of the present disclosure;

[0018] Figure 3is a block diagram illustrating an example of a structure of software (computer program) that may be used in the camera 100 and the controller 200 according to one or more embodiments of the present disclosure;

[0019] Figure 4A is a flowchart of one or more operations that may be used for camera 100 according to one or more embodiments of the present disclosure;

[0020] Figure 4B is a flow chart of one or more operations that may be used for the controller 200 according to one or more embodiments of the present disclosure;

[0021] Figure 5A is a graph formed based on the relationship between distance difference and angular rate according to one or more embodiments of the present disclosure;

[0022] Figure 5B is a graph showing a corresponding relationship between distance difference and angular rate according to one or more embodiments of the present disclosure;

[0023] Figure 6A is a diagram illustrating a display example of a captured image according to one or more embodiments of the present disclosure;

[0024] Figure 6B is a diagram showing a state in which a user sets a new target location according to one or more embodiments of the present disclosure;

[0025] Figure 7A is a block diagram illustrating an example of a hardware structure that may be used for the camera 100 according to one or more embodiments of the present disclosure;

[0026] Figure 7B is a block diagram illustrating an example of the structure of software (computer program) that may be used in the camera 100 according to one or more embodiments of the present disclosure;

[0027] Figure 8A-1 and Figure 8A-2 is a flowchart of one or more operations that may be used for camera 100 according to one or more embodiments of the present disclosure;

[0028] Figure 8B is a flow chart of one or more operations that may be used for the controller 200 according to one or more embodiments of the present disclosure;

[0029] Figure 9A is a graph formed according to the relationship between size difference and zoom speed according to one or more embodiments of the present disclosure; and

[0030] Figure 9Bis a graph showing a corresponding relationship between size difference and zoom speed according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0031] Hereinafter, one or more embodiments and / or features of the present disclosure will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of protection claimed by the present disclosure. A plurality of features are described in the embodiments, but no limitation is made to the embodiments requiring all such features, and a plurality of such features may be appropriately combined. The structures in the following embodiments are merely examples, and the (one or more) techniques / (one or more) features present in the present disclosure are not limited to the illustrated structures. In addition, in the accompanying drawings, the same figure numerals are given to the same or similar structures, and their redundant descriptions are omitted.

[0032] The following are details of one or more embodiments of the present disclosure:

[0033] First, refer to Figure 1 To describe an example of the structure of one or more system embodiments of the present disclosure. Figure 1 As shown, at least one system may include a camera 100 and a controller 200 serving as a control device for the camera 100. The camera 100 and the controller 200 are connected to a network 300. The system according to one or more embodiments is configured or operated to perform data communication between the camera 100 and the controller 200 via the network 300. The network 300 includes networks such as a local area network (LAN) and the Internet.

[0034] Next, we will refer to Figure 2 1 and 2. An example of a hardware structure that can be used for the camera 100 and the controller 200 is described as a block diagram. Note that Figure 2 The illustrated structure is merely an example of the hardware structure of the camera 100 and the controller 200 for one or more embodiments, and the hardware structure may be appropriately changed / modified for one or more additional embodiments.

[0035] First, an example of at least one hardware structure of the camera 100 will be described. The camera 100 has a mechanism capable of performing pan / tilt operations to change the posture (imaging direction) of the device itself, and changes the posture by performing pan / tilt operations based on the result of detecting a subject from a captured image.

[0036] The CPU 101 executes various processes using computer programs and data stored in the RAM 102. The CPU 101 controls the overall operation of the camera 100, and executes or controls various processes described as processes to be performed by the camera 100.

[0037] In one or more embodiments, the RAM 102 is a high-speed storage device such as a DRAM. The RAM 102 includes an area for storing computer programs and data loaded from the storage device 103 and an area for storing captured images output from the image processing unit 106. Furthermore, the RAM 102 includes an area for storing various types of information received from the controller 200 via the network I / F 105 and a work area used when the CPU 101 and the inference unit 104 execute various processes. In this manner, the RAM 102 can provide various storage or work areas as appropriate.

[0038] The storage device 103 is a nonvolatile storage device such as a flash memory, a hard disk drive (HDD), a solid-state drive (SSD), or a secure digital (SD) card. The storage device 103 stores setting data for the camera 100, computer programs and data related to activation of the camera 100, computer programs and data related to basic operations of the camera 100, and the like. The storage device 103 may also store computer programs and data for causing the CPU 101 and the inference unit 104 to execute or control various processes described as being to be performed by the camera 100.

[0039] Inference unit 104 performs inference processing for inferring the presence / absence and position of a subject from captured images. Inference unit 104 is, for example, a computing device such as a graphics processing unit (GPU) dedicated to image processing and inference processing. GPUs are often effectively used for inference processing, but similar functions can also be implemented using reconfigurable logic circuits such as field programmable gate arrays (FPGAs). The processing of inference unit 104 can be performed by CPU 101.

[0040] The network I / F 105 is an interface for connecting to the network 300 and is connected to the network 300 via Ethernet. Communication with an external device such as the controller 200 is performed using a communication medium such as . . . Note that a serial communication I / F may be prepared separately and used for communication.

[0041] The image processing unit 106 converts the image signal output from the image sensor 107 into a captured image as data in a predetermined format, compresses the captured image if necessary, and then outputs it to the RAM 102. Note that the image processing unit 106 can perform various processes on the image represented by the image signal obtained from the image sensor 107, such as image quality adjustment including color correction, exposure correction, and sharpness correction, and cropping processing to crop an image into only a predetermined area. These processes can be executed according to instructions received from the controller 200 via the network I / F 105.

[0042] The image sensor 107 receives light reflected from a subject, converts the brightness and color of the received light into electric charge, and outputs an image signal based on the conversion result. As the image sensor 107, for example, a photodiode, a charge coupled device (CCD) sensor, or a complementary metal oxide semiconductor (CMOS) sensor can be used.

[0043] The drive I / F 108 is an interface for transmitting / receiving a command signal such as a control signal to / from the drive unit 109 .

[0044] The drive unit 109 is a drive mechanism for changing the posture of the camera 100, and includes a mechanical drive system and a drive source motor. In accordance with instructions received from the CPU 101 via the drive I / F 108, the drive unit 109 performs a pan / tilt operation for changing the posture of the camera 100 horizontally and vertically, and a zoom operation for optically changing the angle of view.

[0045] The CPU 101 , the RAM 102 , the storage device 103 , the inference unit 104 , the network I / F 105 , the image processing unit 106 , and the drive I / F 108 are all connected to a system bus 110 .

[0046] Next, the controller 200 will be described. The controller 200 can receive captured images transmitted from the camera 100 via the network 300, and transmit a target position of a tracking object based on a user's operation on the controller 200 to the camera 100. With this system, the user can use the controller 200 to specify a target position of a tracking object, and use the camera 100 to track the tracking object and capture an image of the tracking object so that the tracking object approaches the selected or specified target position.

[0047] The CPU 201 executes various processes using computer programs and data stored in the RAM 202. The CPU 201 controls the overall operation of the controller 200, and executes or controls various processes described as processes to be performed by the controller 200.

[0048] In one or more embodiments, the RAM 202 is a high-speed storage device such as a DRAM. The RAM 202 includes an area for storing computer programs and data loaded from the storage device 203 and an area for storing various data received from the camera 100 via the network interface 205. Furthermore, the RAM 202 includes a work area used when the CPU 201 and the inference unit 204 execute various processes. In this manner, the RAM 202 can provide various storage or work areas as appropriate.

[0049] The storage device 203 is a non-volatile storage device such as a flash memory, an HDD, an SSD, or an SD card. The storage device 203 stores setting data for the controller 200, computer programs and data related to the startup of the controller 200, computer programs and data related to the basic operation of the controller 200, and the like. The storage device 203 may also store computer programs and data for causing the CPU 201 and the inference unit 204 to execute or control the various processes described as being to be performed by the controller 200.

[0050] The inference unit 204 performs inference processing for inferring the presence / absence and position of a subject from the captured image. The inference unit 204 is, for example, a computing device such as a graphics processing unit (GPU) dedicated to image processing and inference processing. GPUs are often effectively used for inference processing, but similar functions can also be implemented using reconfigurable logic circuits such as field programmable gate arrays (FPGAs). The processing of the inference unit 204 can be performed by the CPU 201.

[0051] The network I / F 205 is an interface for connecting to the network 300 and communicates with an external device such as the camera 100 via a communication medium such as Ethernet. For example, communication with the camera 100 includes sending control commands to the camera 100 and receiving captured images from the camera 100.

[0052] The display unit 206 is a display unit having a screen such as a liquid crystal screen or a touch panel screen. The display unit 206 can display captured images received from the camera 100 and a setting screen of the controller 200. In at least this embodiment, the case where the display unit 206 is a display unit having a touch panel screen will be described.

[0053] Note that the features and one or more embodiments of the present disclosure are not limited to the case where the controller 200 includes the display unit 206. For example, the display unit 206 may be omitted from the controller 200, and a display device may be connected to the controller 200 to display a captured image and a setting screen of the controller 200, etc. on the display device.

[0054] The user input I / F 207 is an interface for accepting an operation on the controller 200 from a user, and includes, for example, a button, a dial, a joystick, and a touch panel.

[0055] The CPU 201, RAM 202, storage device 203, inference unit 204, network I / F 205, display unit 206, and user input I / F 207 are all connected to a system bus 208. Note that the controller 200 may be a personal computer (PC) having a mouse and keyboard as the user input I / F 207.

[0056] Next, Figure 3 1 is a block diagram showing an example of the structure of software (computer program) that can be used in the camera 100 and the controller 200. Note that Figure 3 Illustration of general software such as an operating system is omitted.

[0057] In this embodiment, the camera 100 stores, as software components, an imaging module 301, an inference module 302, a drive control module 303, an arithmetic operation module 304, and a communication module 305 in the storage device 103. The CPU 101 appropriately deploys these software components from the storage device 103 to the RAM 102 and uses them.

[0058] In this embodiment, the controller 200 stores a user interface module 306, an inference module 307, an arithmetic operation module 308, and a communication module 309 as software components in the storage device 203. The CPU 201 appropriately deploys these software components from the storage device 203 to the RAM 202 and uses them.

[0059] Notice, Figure 3 The software structure shown in is only an example. For example, a functional unit can be divided into multiple functional units according to function, or multiple functional units can be integrated into one functional unit. It can be implemented by hardware. Figure 3 One or more functional units are shown.

[0060] Next, the respective operations of the camera 100 and the controller 200 in the system according to the present embodiment will be described. Figure 4A The operation of the camera 100 is described with reference to a flowchart of FIG.

[0061] In step S401, the CPU 101 reads out the camera module 301 from the storage device 103 to the RAM 102, deploys the camera module 301, and executes the deployed camera module 301. The CPU 101 then obtains a captured image from the image processing unit 106 and stores it in the RAM 102.

[0062] In step S402, the CPU 101 reads the communication module 305 from the storage device 103 to the RAM 102, deploys it in the RAM 102, and executes the deployed communication module 305. The CPU 101 then transmits the captured image stored in the RAM 102 to the controller 200 via the network I / F 105.

[0063] In step S403, the CPU 101 reads the arithmetic operation module 304 from the storage device 103 to the RAM 102, deploys it in the RAM 102, and executes the deployed arithmetic operation module 304. The CPU 101 then determines whether the “target position of the tracking subject” transmitted from the controller 200 has been received via the network I / F 105.

[0064] If the CPU 101 determines that the target position has been received, it shifts the process to step S404. If the CPU 101 determines that the target position has not been received, it shifts the process to step S405.

[0065] In step S404, the CPU 101 executes the arithmetic operation module 304 disposed in the RAM 102. Then, the CPU 101 stores the target position received from the controller 200 via the network I / F 105 in the RAM 102.

[0066] In step S405, the CPU 101 reads the inference module 302 from the storage device 103 to the RAM 102, deploys it in the RAM 102, and executes the deployed inference module 302. Then, the CPU 101 inputs the captured image stored in the RAM 102 in step S401 to the inference unit 104, controls the inference unit 104 to detect the tracking subject from the captured image, and stores the result of the detection (hereinafter referred to as the detection result) in the RAM 102.

[0067] At this point, the inference unit 104 reads the learned model, created using a machine learning method such as deep learning, from the storage device 103 to the RAM 102 and stores it in the RAM 102. The inference unit 104 then detects the tracking subject from the captured image by inputting the captured image into the learned model and performing computational processing on the learned model. The inference unit 104 then outputs positional information indicating the position of the tracking subject in the captured image as a detection result. Note that the CPU 101 can reduce the captured image, and the inference unit 104 can input the reduced captured image into the learned model. This can reduce the processing load of the inference unit 104 and accelerate the inference processing.

[0068] The results of inference unit 104 detecting a tracked subject will now be described. If inference unit 104 inputs a captured image into a learned model and performs computational processing based on the learned model, inference unit 104 outputs rectangle information (e.g., the coordinates of the upper left and lower right vertices of the rectangle) defining a rectangle encompassing the entire body of the tracked subject as positional information indicating the position of the tracked subject in the captured image. Note that rectangle information is not limited to information representing the entire body of the tracked subject; it can also be information representing a portion of the tracked subject. For example, if the tracked subject is a person, this information can be the position of the person's head or face. In this case, the learned model used can be modified to provide the desired input / output. Furthermore, the positional information of the tracked subject is not limited to the coordinates of the upper left and lower right vertices of the rectangle encompassing the entire body of the tracked subject; information that defines the position of the tracked subject in the captured image (such as the coordinates of the center of the rectangle or the width or height of the rectangle) is sufficient.

[0069] Note that the method by which the inference unit 104 detects the tracking subject from the captured image is not limited to a specific method. For example, the inference unit 104 may use a template matching method in which a template image of the tracking subject is registered in advance in the storage device 103 or the like, and an area in the captured image that is highly similar to the template image is detected as the tracking subject area.

[0070] In step S406, the CPU 101 executes the arithmetic operation module 304 disposed in the RAM 102. Then, the CPU 101 calculates the distance (distance difference) between the target position stored in the RAM 102 and the subject position derived from the detection result of the tracking subject stored in the RAM 102. For example, if the target position is the center position of the tracking subject area, the CPU 101 calculates the center position of the tracking subject area as the subject position based on the detection result of the tracking subject stored in the RAM 102.

[0071] The distance difference indicates the distance from the target position to the subject position on the captured image and is expressed in pixels. The CPU 101 stores the calculated distance difference in the RAM 102 .

[0072] In step S407, the CPU 101 executes the arithmetic operation module 304 disposed in the RAM 102. Then, the CPU 101 determines whether the distance difference stored in the RAM 102 is smaller than a second threshold. The second threshold is a threshold value preset as a standard for determining whether the distance difference between the target position and the subject position is sufficiently small.

[0073] If the CPU 101 determines that the distance difference stored in the RAM 102 is less than the second threshold, it transfers the process to step S410. If the CPU 101 determines that the distance difference stored in the RAM 102 is not less than the second threshold, it transfers the process to step S408.

[0074] Note that the branching condition in step S407 is not limited to the threshold determination based on the distance difference. Since the CPU 101 stops the tracking operation when the distance difference is sufficiently small, the CPU 101 can branch the processing after step S407 based on whether the tracking operation has stopped. For example, whether the tracking operation has stopped can be determined by whether the angle sensor (not shown) measuring the rotation angle of the drive unit 109 has stopped changing its angle. If the tracking operation has stopped, the CPU 101 transfers the processing to step S410. If the tracking operation has not stopped, the CPU 101 transfers the processing to step S408.

[0075] As another example, the processing from step S407 can be branched based on the time elapsed after switching to the speed calculation of formula (4) to be described later. In this case, the CPU 101 measures the time elapsed after switching to the speed calculation of formula (4). If the elapsed time is equal to or greater than the threshold value, the CPU 101 transfers the processing to step S410. If the elapsed time is less than the threshold value, the CPU 101 transfers the processing to step S408.

[0076] In step S408, the CPU 101 executes the arithmetic operation module 304 disposed in the RAM 102. Then, the CPU 101 determines whether the distance difference stored in the RAM 102 is smaller than a first threshold value, wherein the first threshold value is larger than a second threshold value.

[0077] The first threshold is a preset threshold used to determine whether the tracked subject is merely oscillating in place or has begun to move. That is, in this embodiment, if the distance difference between the target position and the subject position is less than the first threshold, the tracked subject is determined to be merely oscillating in place. If the distance difference is not less than the first threshold, the tracked subject is determined to have moved.

[0078] Note that the first threshold is set by reading a value pre-stored in RAM 102 or storage device 103. However, since the relative distance difference varies depending on the image size of the tracked object, the first threshold can be set to an optimal threshold corresponding to the image size. This allows for handling situations where the zoom changes during tracking.

[0079] If the CPU 101 determines that the distance difference stored in the RAM 102 is smaller than the first threshold, it transfers the process to step S411. If the CPU 101 determines that the distance difference stored in the RAM 102 is not smaller than the first threshold, it transfers the process to step S409.

[0080] In step S409, the CPU 101 sets the value of the threshold value exceeding flag (a flag for storing the result that the distance difference stored in the RAM 102 exceeds the first threshold value) to ON, stores it in the RAM 102, and then transfers the processing to step S412. On the other hand, in step S410, the CPU 101 sets the value of the threshold value exceeding flag to OFF, stores it in the RAM 102, and then transfers the processing to step S412. The initial value of the threshold value exceeding flag is set to "OFF".

[0081] In step S411, the CPU 101 executes the arithmetic operation module 304 disposed in the RAM 102. Then, the CPU 101 determines whether the value of the threshold-exceeding flag stored in the RAM 102 is ON.

[0082] If the CPU 101 determines that the value of the threshold-exceeding flag is ON, it shifts the process to step S413. If the CPU 101 determines that the value of the threshold-exceeding flag is OFF, it shifts the process to step S412.

[0083] In step S412, the CPU 101 executes the arithmetic operation module 304 deployed in the RAM 102. Then, the CPU 101 converts the distance difference stored in the RAM 102 into an angle difference (the pan angle and the pitch angle of the camera 100). For example, the CPU 101 uses the information of the imaging resolution and the imaging angle of view of the camera 100 to approximately calculate the angle of each pixel of the captured image, and calculates the result of multiplying the angle by the distance difference as the angle difference. Then, the CPU 101 calculates the angular rate in the pan / tilt direction corresponding to the calculated angle difference. Assuming that a is the calculated angle difference, g_1 is the first rate coefficient, and v_1 is the first offset angular rate, the CPU 101 calculates the first angular rate ω_1 according to formula (1):

[0084] ω_1=a×g_1+v_1...(1)

[0085] Figure 5A A graph is shown which shows the relationship between the distance difference and the angular rate. Figure 5AIn the graph of , the abscissa represents the distance difference, and the ordinate represents the angular rate. The straight line obtained by the above formula (1) is the straight line 504, and the straight line 504 has a shape such that the angular rate increases in proportion to the distance difference. The distance difference 502 indicates the first dead zone associated with the straight line 504 of the formula (1). The dead zone indicates the range of the distance difference in which the angular rate in the pan direction and the angular rate in the pitch direction are 0. That is, the straight line 504 of the formula (1) outputs a negative angular rate within a range less than the distance difference 502, but in the case where the distance difference is less than the distance difference 502, the CPU 101 outputs 0 as the angular rate corresponding to the straight line 504.

[0086] Note that even for the same distance difference, as the value of the first velocity coefficient g_1 increases, the corresponding angular velocity increases, and as the value of the first offset angular velocity v_1 decreases, the first dead zone increases. The values of the first velocity coefficient g_1 and the first offset angular velocity v_1 can be determined through experimentation or can be arbitrarily set by the user by operating the controller 200.

[0087] Then, the CPU 101 generates a control command for causing the drive unit 109 to rotate the camera 100 in the pan direction and / or the tilt direction at a first angular velocity therein, and stores the generated control command in the RAM 102 .

[0088] In step S413, the CPU 101 executes the arithmetic operation module 304 disposed in the RAM 102. Then, similarly to step S412, the CPU 101 calculates the angle difference (the pan angle and the pitch angle of the camera 100) from the distance difference stored in the RAM 102. The CPU 101 then calculates the angular rate in the pan / tilt direction corresponding to the calculated angle difference. Assuming that a is the calculated angle difference, g_2 is the second rate coefficient, and v_2 is the second offset angular rate, the CPU 101 calculates the second angular rate ω_2 according to formula (2):

[0089] ω_2=a×g_2+v_2…(2)

[0090] exist Figure 5A In the equation (2), the straight line obtained by the above equation is straight line 505, and similar to the above equation (1), straight line 505 has a shape such that the angular rate increases in proportion to the distance difference. Distance difference 501 indicates a second dead zone associated with straight line 505 of equation (2). That is, straight line 505 of equation (2) outputs a negative angular rate within a range less than distance difference 501, but when the distance difference is less than distance difference 501, CPU 101 outputs 0 as the angular rate corresponding to straight line 505.

[0091] Note that even for the same distance difference, as the value of the second rate coefficient g_2 increases, the corresponding angular rate increases, and as the value of the second offset angular rate v_2 decreases, the second dead zone increases. The values of the second rate coefficient g_2 and the second offset angular rate v_2 can be determined through experiments or can be arbitrarily set by the user by operating the controller 200. Note that the first rate coefficient g_1, the second rate coefficient g_2, the first offset angular rate v_1, and the second offset angular rate v_2 are determined such that: within the range from the distance difference 501 (including the distance difference 501) to the distance difference 503 (excluding the distance difference 503), the angular rate corresponding to the distance difference X within this range on the straight line 505 of formula (2) is greater than the angular rate corresponding to the distance difference X on the straight line 504 of formula (1). For example, these values are set to satisfy g_2 < g_1 and v_2 > v_1.

[0092] Then, the CPU 101 generates a control command for rotating the camera 100 in the pan direction and / or the tilt direction at a second angular rate in the pan direction and / or the tilt direction, and stores the generated control command in the RAM 102.

[0093] Taking the above points into consideration, Figure 5B shows the correspondence between the distance difference and the angular rate. In this embodiment, the distance difference 501 (using this distance difference 501, since the distance difference from the target position to the subject position is small enough, the tracking operation stops) is set as the second threshold, and the distance difference 503 (using this distance difference 503, since the distance difference from the target position to the subject position is large, it can be determined that the tracked subject has moved) is set as the first threshold.

[0094] As Figure 5B shown, the straight line 504 is substantially a function as follows: when the distance difference is less than the distance difference 502, output 0 as the first angular rate, and when the distance difference is equal to or greater than the distance difference 502, output the first angular rate calculated according to formula (1). In addition, the straight line 505 is substantially a function as follows: when the distance difference is less than the distance difference 501, output 0 as the second angular rate, and when the distance difference is equal to or greater than the distance difference 501 and less than the distance difference 503, output the second angular rate calculated according to the above formula (2).

[0095] Note that, regarding the method of calculating the angular rate when the distance difference is equal to or greater than the distance difference 503, there is no limitation on which of formula (1) and formula (2) is used in one or more embodiments, but this embodiment assumes that the angular rate is calculated by formula (1). This is because the result of formula (1) is greater than the result of formula (2) for the angular rate in the pan direction and the pitch direction when the distance difference is equal to or greater than the distance difference 503, and therefore it is easy to maintain tracking even when tracking a subject at high speed.

[0096] Therefore, in Figure 5B In the example shown, if the distance difference is less than the distance difference 501 , the angular rate is 0, and if the distance difference is equal to or greater than the distance difference 503 , the angular rate is the first angular rate calculated by the above formula (1).

[0097] If the distance difference changes from the state of "distance difference is less than distance difference 501" to the state of "distance difference is equal to or greater than distance difference 501 and less than distance difference 503", the angular rate is the first angular rate calculated by the above formula (1) (but in the case where the distance difference is equal to or greater than distance difference 501 and less than distance difference 502, the angular rate is 0 due to the dead zone processing).

[0098] If the distance difference changes from the state of "distance difference equal to or greater than distance difference 503" to the state of "distance difference equal to or greater than distance difference 501 and less than distance difference 503", the angular rate is the second angular rate calculated by the above formula (2).

[0099] Note that in this implementation example, Figure 5B As shown, the distance difference at the intersection of straight line 504 and straight line 505 is set to distance difference 503, and if the distance difference exceeds distance difference 503, the straight line used is switched between straight line 504 and straight line 505. Therefore, if the distance difference is distance difference 503, the angular rate between straight line 504 and straight line 505 is consistent. Therefore, even in the case where distance difference 503 is set to the first threshold and the straight line used is switched from straight line 504 to straight line 505, the output angular rate can be prevented from being discontinuous in time. However, distance difference 503 is not limited to this and can be greater than or less than the value at the intersection. In this case, straight line 504 gradually switches to straight line 505 to prevent the output angular rate from being discontinuous in time.

[0100] By using the processing in steps S407 to S413, for example, at the start of the tracking operation, the tracking operation is slowly started at a small angular rate in the pan / tilt direction obtained by formula (1). Next, during tracking, the posture of the camera 100 is controlled to reduce the distance difference, and as the distance difference increases, a larger angular rate in the pan / tilt direction given by formula (1) is obtained. Then, when the tracking operation is stopped after the distance difference exceeds the first threshold, the control method is switched, and a larger angular rate in the pan / tilt direction given by formula (2) is obtained to further reduce the distance difference. As a result, it becomes easier to bring the subject position closer to the target position.

[0101] Note that this embodiment has described a method for obtaining the linear angular rate given by Formula (1) or Formula (2) as a method for calculating the control method, but the calculation method is not limited to this. For example, two types of quadratic curves having a relationship similar to the above Formula (1) and Formula (2) can be used to obtain the angular rate. This allows the tracking operation to start slowly and allows the tracking operation to be maintained more easily even if the distance difference increases during tracking.

[0102] In step S414, the CPU 101 reads the drive control module 303 from the storage device 103 to the RAM 102, stores it in the RAM 102, and executes the stored drive control module 303. The CPU 101 then derives drive parameters for performing pan / tilt operations in desired directions and at desired angular rates from the control commands stored in the RAM 102. The drive parameters indicate parameters for driving and controlling motors (not shown) included in the drive unit 109 in each of the pan and tilt directions. The CPU 101 then drives and controls the drive unit 109 via the drive I / F 108 based on the derived drive parameters. When the drive unit 109 is driven based on the drive parameters, the camera 100 changes its imaging direction (posture), i.e., performs a pan / tilt operation.

[0103] In step S415, the CPU 101 executes the arithmetic operation module 304 deployed in the RAM 102. The CPU 101 then determines whether the termination condition for terminating the tracking is satisfied. Various conditions may apply to the termination condition, and the features and one or more embodiments of the present disclosure are not limited to specific conditions. Examples of the termination condition include a condition that "a tracking termination instruction has been received from the controller 200," a condition that "the current date / time has reached a predetermined date / time," or a condition that "a predetermined time has passed since the start of tracking."

[0104] If the CPU 101 determines that the termination condition is satisfied, it terminates the process according to Figure 4AOn the other hand, if the CPU 101 determines that the end condition is not satisfied, it returns the process to step S401.

[0105] Next, we will refer to Figure 4B The flowchart shown is used to illustrate the operation of the controller 200.

[0106] In step S416, the CPU 201 reads the arithmetic operation module 308 from the storage device 203 to the RAM 202, stores it in the RAM 202, and executes the arithmetic operation module 308. The CPU 201 then determines whether a captured image transmitted from the camera 100 has been received via the network I / F 205. Note that the captured image can be received via the video input I / F (not shown) of the controller 200 instead of the network I / F 205. In this case, the video input I / F is connected to the video output I / F (not shown) of the camera 100 via a video transmission cable. This can cope with a situation where the communication bandwidth of the network 300 is insufficient when transmitting / receiving captured images.

[0107] If the CPU 201 determines that a captured image has been received from the camera 100, it stores the received captured image in the RAM 202 and advances the process to step S417. On the other hand, if the CPU 201 determines that a captured image has not been received from the camera 100, it returns the process to step S416.

[0108] In step S417, the CPU 201 reads out the user interface module 306 from the storage device 203 to the RAM 202, deploys it in the RAM 202, and executes the deployed user interface module 306. The CPU 201 then displays the captured image stored in the RAM 202 on the display unit 206.

[0109] Figure 6A 1 shows a display example of a captured image on the display unit 206. Figure 6A As shown, a captured image including a tracking subject 602 is displayed on the display screen of the display unit 206, and an icon 601 indicating the current target position when imaging the tracking subject 602 is superimposed and displayed on the captured image. The user can confirm the image captured by the camera 100 and the target position by checking the display screen of the display unit 206.

[0110] In step S418, the CPU 201 executes the user interface module 306 disposed in the RAM 202. Then, the CPU 201 accepts a touch operation for “tracking subject target position setting operation” on the display unit 206 by the user. Figure 6B Shown in Figure 6AThe example shows the state where the user sets a new target location. Figure 6B 6. In FIG. 6, the user touches a position different from the target position indicated by icon 601 with his / her finger, thereby setting the position as a new target position. Icon 603 indicates the target position newly set by the user operation.

[0111] Note that the method of setting the target position is not limited to a specific method. For example, the user can set the target position by operating the user input I / F 207. Then, the CPU 201 determines whether a touch operation for "tracking subject target position setting operation" has been input.

[0112] If the CPU 201 determines that a touch operation for “setting a tracking subject target position” has been input, it stores the position on the captured image set by the touch operation as a new target position in the RAM 202 and transfers the process to step S419. On the other hand, if the CPU 201 determines that a touch operation for “setting a tracking subject target position” has not been input, it returns the process to step S416.

[0113] Note that in this embodiment, the target for tracking a subject is "position," but this is not the only option. For example, a "region" (target area) can be set as the target for tracking a subject. Within the target area, if the subject's position enters the target area, it is determined that the subject has reached the target position. This is effective because it can suppress excessive tracking operations caused by small fluctuations in the subject's position.

[0114] In step S419, the CPU 201 reads the communication module 309 from the storage device 203 to the RAM 202, deploys it in the RAM 202, and executes the communication module 309. The CPU 201 then transmits the new target position stored in the RAM 202 to the camera 100 via the network I / F 205.

[0115] As described above, in this embodiment, the movement / non-movement of the subject is determined based on the distance difference between the target position and the subject position, and the method for controlling the posture of the camera 100 is switched based on the determination result. Therefore, in the case where the subject moves, the subject position can be easily brought closer to the target position.

[0116] That is, in the present embodiment, in a structure for calculating a control speed for controlling the posture of a camera device so that a detection position of a subject detected in an image captured by the camera device becomes closer to a target position of the subject in the captured image, an example of a camera device has been described as follows: in a state where a difference between the detection position and the target position is equal to or greater than a first threshold value, a first control speed is calculated as the control speed of the posture based on the difference, and in a state where the difference becomes less than the first threshold value and equal to or greater than a second threshold value which is smaller than the first threshold value after the above state, a second control speed higher than the first control speed is calculated as the control speed of the posture based on the difference.

[0117] Note that as will be described below, any features of the present disclosure of (one or more) embodiments and (one or more) embodiments discussed below can be combined. In this case, when the control speed of the posture of the camera device is less than the threshold value, in a state where the above difference is equal to or greater than the first threshold value, the first control speed can be calculated as the control speed of the posture based on the difference, and when the difference becomes less than the first threshold value and equal to or greater than the second threshold value after the above state, the second control speed can be calculated as the control speed of the posture based on the difference.

[0118] When the control speed of the posture of the camera device is equal to or greater than a threshold, the first control speed can be calculated as the control speed of the posture based on the difference, and when the difference becomes less than the first threshold and equal to or greater than the second threshold after the difference is less than the second threshold, the control speed of the posture can be set to 0.

[0119] Note that in this embodiment, the distance difference at the time the captured image is obtained is calculated based on the captured image obtained in step S401. However, the method for calculating the distance difference is not limited to this. For example, the distance difference at that time may be calculated by accumulating the distance differences calculated for each captured image up to that time. Therefore, since the movement speed of the tracked subject is balanced with the pan / tilt angular rate of camera 100, even if the distance difference between the target position and the subject position on the captured image remains unchanged, it is possible to determine whether the tracked subject has moved based on the cumulative distance difference.

[0120] Note that in this embodiment, the camera 100 calculates the driving amount for detecting and tracking the subject. However, the controller 200 may perform some or all of the processing. In this case, for example, the camera 100 transmits a captured image to the controller 200. The controller 200 detects the tracking subject from the received captured image, calculates driving parameters for tracking the tracking subject (as described above), and transmits the driving parameters to the camera 100. The camera 100 then tracks the tracking subject based on the received driving parameters (as described above) and captures an image of the tracking subject. In this case, the processing performed by the inference unit 104 of the camera 100 may be performed by the inference unit 204 of the controller 200, and the functions of the inference module 302 and the arithmetic operation module 304 of the camera 100 may be performed by the inference module 307 and the arithmetic operation module 308 of the controller 200, respectively. Therefore, even if the camera 100 does not have an inference function, the same effect can be achieved.

[0121] In one or more additional embodiments, the differences from the (one or more) embodiments described above will be described below, and unless otherwise specified, the one or more additional embodiments are similar to the (one or more) embodiments described above. The one or more embodiments discussed below will describe a method for switching the method for calculating the angular velocity based on a comparison of a threshold value with the difference between the target size of the tracked subject in the imaging composition and the detected size of the tracked subject (hereinafter referred to as the subject size), thereby making it easy to bring the subject size closer to the target size. Note that the control to be described in the (one or more) present embodiments and the control described in the (one or more) embodiments described above may be performed in combination in one or more embodiments.

[0122] (One or more) This embodiment will also describe an example in which a new structure is provided for measuring the operating speed of the drive unit 109 of the camera 100, and the method for calculating the angular rate is switched according to the operating speed. Note that switching the method for calculating the angular rate according to the operating speed can be performed in conjunction with the control described above.

[0123] Will refer to Figure 7A 1 and 2 to describe an example of at least one hardware structure of the camera 100 according to one or more embodiments. Figure 7A The illustrated structure is merely an example of the hardware structure of the camera 100 and can be changed / modified as appropriate.

[0124] The speed sensor 701 is a sensor for measuring the operating speed (drive speed) of the drive mechanism provided in the drive unit 109 of the camera 100. In the present embodiment(s), the speed sensor 701 obtains the drive speed of the drive unit 109 in degrees per second. The speed sensor 701 is connected to the system bus 110.

[0125] Next, Figure 7B is a block diagram showing an example of at least one embodiment of the structure of software (computer program) in the camera 100. Note that Figure 7B The illustration of common software such as operating system is omitted. Figure 7B The software structure shown is only an example. For example, a functional unit can be divided into multiple functional units according to function, or multiple functional units can be integrated into one functional unit. It can be implemented by hardware. Figure 7B One or more functional units are shown.

[0126] Next, the respective operations of the camera 100 and the controller 200 in the system according to one or more embodiments will be described. Figure 8A-1 and 8A-2 The operation of the camera 100 will be described with reference to a flowchart of FIG.

[0127] Step S801 and Figure 4A The process is the same as step S401, and its description will be omitted.

[0128] In step S802, the CPU 101 reads the arithmetic operation module 304 from the storage device 103 to the RAM 102, deploys it in the RAM 102, and executes the deployed arithmetic operation module 304. The CPU 101 then determines whether the target size of the tracking object has been received from the controller 200 via the network I / F 105.

[0129] If the CPU 101 determines that the target size has been received, it shifts the process to step S803. If the CPU 101 determines that the target size has not been received, it shifts the process to step S804.

[0130] The target size of the tracking subject is the size of the tracking subject to be captured in the captured image, and in (one or more than one) present embodiments, the target size is represented by the ratio of the length of the tracking subject area in the horizontal direction to the length of the captured image in the horizontal direction. That is, the target size is represented by the ratio of the width of the tracking subject to the width of the captured image. Note that the target size is not limited to the above-mentioned width ratio and can be the ratio of the height of the tracking subject to the height of the captured image. That is, the method for calculating the size of the tracking subject relative to the captured image is not limited to a specific method. In addition, the target size is defined in advance, and, for example, a ratio is defined for capturing the size of the entire body or upper body of the tracking subject in the captured image.

[0131] In step S803, the CPU 101 executes the arithmetic operation module 304 disposed in the RAM 102. Then, the CPU 101 stores in the RAM 102 the target size received from the controller 200 via the network I / F 105.

[0132] Step S804 and Figure 4A However, in this embodiment, size information is output as the detection result, and the size information indicates the length of the tracking object region in the captured image in the horizontal direction.

[0133] In step S805, the CPU 101 executes the arithmetic operation module 304 deployed in the RAM 102. The CPU 101 then calculates the ratio of the "length indicated by the size information" to the horizontal length of the captured image as the subject size. The CPU 101 calculates the difference (size difference) between the subject size and the target size stored in the RAM 102 and stores the size difference in the RAM 102.

[0134] In step S806, the CPU 101 reads the measurement module 702 from the storage device 103 to the RAM 102, deploys it in the RAM 102, and executes the deployed measurement module 702. Then, the CPU 101 measures the driving speed of the driving unit 109 and stores the measured driving speed in the RAM 102.

[0135] In step S807, the CPU 101 executes the arithmetic operation module 304 disposed in the RAM 102. The CPU 101 then determines whether the size difference stored in the RAM 102 is smaller than a fourth threshold. The fourth threshold is a threshold value preset as a standard for determining whether the size difference between the target size and the subject size is sufficiently small.

[0136] If the CPU 101 determines that the size difference is smaller than the fourth threshold, it shifts the process to step S811. On the other hand, if the CPU 101 determines that the size difference is not smaller than the fourth threshold, it shifts the process to step S808.

[0137] In step S808, the CPU 101 executes the arithmetic operation module 304 disposed in the RAM 102. Then, the CPU 101 determines whether the driving speed of the driving unit 109 stored in the RAM 102 is less than a fifth threshold value.

[0138] If the CPU 101 determines that the driving speed is less than the fifth threshold, it shifts the process to step S809. If the CPU 101 determines that the driving speed is not less than the fifth threshold, it shifts the process to step S813.

[0139] In step S809, the CPU 101 executes the arithmetic operation module 304 stored in the RAM 102. The CPU 101 then determines whether the size difference stored in the RAM 102 is less than a third threshold, where the third threshold is greater than a fourth threshold. The third threshold is a threshold value preset as a standard for determining whether the tracked subject is merely oscillating in place or has begun to move.

[0140] If the CPU 101 determines that the size difference stored in the RAM 102 is smaller than the third threshold, it transfers the process to step S812. If the CPU 101 determines that the size difference stored in the RAM 102 is not smaller than the third threshold, it transfers the process to step S810.

[0141] That is, in this embodiment, if the size difference is less than the third threshold, it is determined that the tracked object is only swinging in place. If the size difference is not less than the third threshold, it is determined that the tracked object has moved.

[0142] Note that the third threshold is set by reading a value pre-stored in RAM 102 or storage device 103. However, since the relative size difference varies depending on the image size of the tracked object, the third threshold can be set to an optimal threshold corresponding to the image size. This allows for handling situations where the zoom changes during tracking.

[0143] In step S810, the CPU 101 sets the value of the threshold value exceeding flag (a flag for storing the result that the size difference stored in the RAM 102 exceeds the third threshold value) to ON, stores it in the RAM 102, and then transfers the processing to step S813. On the other hand, in step S811, the CPU 101 sets the value of the threshold value exceeding flag to OFF, stores it in the RAM 102, and then transfers the processing to step S813. In this embodiment, the initial value of the threshold value exceeding flag is also set to "OFF".

[0144] In step S812, the CPU 101 executes the arithmetic operation module 304 disposed in the RAM 102. Then, the CPU 101 determines whether the value of the threshold-exceeding flag stored in the RAM 102 is ON.

[0145] If the CPU 101 determines that the value of the threshold-exceeding flag is ON, it shifts the process to step S814. If the CPU 101 determines that the value of the threshold-exceeding flag is OFF, it shifts the process to step S813.

[0146] The reason for determining whether the drive speed exceeds a threshold in step S808, in addition to determining whether the size difference exceeds a threshold in step S809, is to accommodate automatic tracking systems with processing delays. For example, in such systems, there is a delay from the time the tracking subject stops until the control command for stopping is applied, and the camera's drive unit continues to drive due to inertia. Therefore, if the drive speed exceeds the threshold, the amount of inertial drive can be expected to increase due to the delay. To address this, even if the high-speed calculation method is selected due to the size difference exceeding the third threshold, the low-speed calculation method is reselected if the drive speed exceeds the fifth threshold. This makes it easier to bring the tracked subject closer to the target position by estimating the amount of inertial drive caused by the delay.

[0147] In step S813, the CPU 101 executes the arithmetic operation module 304 deployed in the RAM 102. Then, the CPU 101 converts the size difference stored in the RAM 102 into an angle difference (the pan angle and the pitch angle of the camera 100). For example, the CPU 101 uses the information of the camera resolution and the camera angle of view of the camera 100 to approximately calculate the angle per 1% of the width of the captured image, and calculates the result of multiplying the size difference by the angle as the angle difference. Then, the CPU 101 calculates the zoom speed as the speed in the zoom direction corresponding to the calculated angle difference. Assuming that a is the calculated angle difference, g_3 is the first speed coefficient, and v_3 is the first offset zoom speed, the CPU 101 calculates the first zoom speed z_1 according to formula (3):

[0148] z_1=a×g_3+v_3...(3)

[0149] Figure 9A A graph is shown which shows the relationship between the size difference and the zoom speed. Figure 9A In the graph of , the horizontal axis represents the size difference, and the vertical axis represents the zoom speed. The straight line obtained by the above formula (3) is straight line 904, and straight line 904 has a shape such that the zoom speed increases in proportion to the size difference. The size difference 902 indicates the first dead zone of the zoom and the third threshold value of the size difference associated with the straight line 904 of formula (3). That is, in the present embodiment, the size difference 902 indicates both the first dead zone of the zoom and the third threshold value of the size difference. The dead zone of the zoom indicates the range of the size difference in which the zoom speed is 0. That is, in the case where the size difference is less than the size difference 902, the CPU 101 outputs 0 as the zoom speed corresponding to the straight line 904.

[0150] Note that even for the same size difference, as the value of the first speed coefficient g_3 increases, the corresponding zoom speed increases, and as the value of the first offset zoom speed v_3 decreases, the first dead zone increases. The values of the first speed coefficient g_3 and the first offset zoom speed v_3 can be determined through experimentation or can be arbitrarily set by the user by operating the controller 200.

[0151] Then, the CPU 101 generates a control command for causing the drive unit 109 to zoom in the zoom direction at the first zoom speed, and stores the generated control command in the RAM 102 .

[0152] In step S814, the CPU 101 executes the arithmetic operation module 304 deployed in the RAM 102. Then, similarly to step S813, the CPU 101 calculates the angle difference (the pan angle and the tilt angle of the camera 100) from the size difference stored in the RAM 102. Then, the CPU 101 calculates the zoom speed corresponding to the calculated angle difference. Assuming that a is the calculated angle difference, g_4 is the second speed coefficient, and v_4 is the second offset zoom speed, the CPU 101 calculates the second zoom speed Z_2 according to formula (4):

[0153] Z_2=a×g_4+v_4...(4)

[0154] exist Figure 9AIn [the above], the straight line obtained through the above formula (4) is the straight line 905, and similar to the above formula (3), the straight line 905 has a shape such that the zoom speed increases proportionally to the size difference. The size difference 901 indicates the second dead zone associated with the straight line 905 of the formula (4). When the size difference is less than the size difference 901, the CPU 101 outputs 0 as the zoom speed corresponding to the straight line 905.

[0155] Figure 9A The zoom speed 903 in the shown graph indicates the fifth threshold of the speed.

[0156] Note that even for the same size difference, as the value of the second speed coefficient g_4 is larger, the corresponding zoom speed is higher, and as the value of the second offset zoom speed v_4 is smaller, the second dead zone is larger. The values of the second speed coefficient g_4 and the second offset zoom speed v_4 can be determined through experiments or can be arbitrarily set by the user by operating the controller 200. Note that the first speed coefficient g_3, the second speed coefficient g_4, the first offset zoom speed v_3, and the second offset zoom speed v_4 are determined such that: within the range of the size difference 901 (including the size difference 901) to the size difference 902 (excluding the size difference 902), the zoom speed corresponding to the distance difference X on the straight line 905 of the formula (4) is higher than the zoom speed corresponding to the distance difference X on the straight line 904 of the formula (3). For example, these values are set to satisfy g_4 < g_3 and v_4 > v_3.

[0157] Then, the CPU 101 generates a control command for causing the drive unit 109 to zoom in the zoom direction at the second zoom speed, and stores the generated control command in the RAM 102.

[0158] Considering the above points, Figure 9B The correspondence between the size difference and the zoom speed is shown. In the present embodiment, the size difference 901 (using which the tracking operation stops because the size difference from the target size to the subject size is small enough) is set as the fourth threshold, and the size difference 902 (using which it can be determined that the tracked subject has moved because the size difference from the target size to the subject size is large) is set as the third threshold.

[0159] As Figure 9BAs shown, straight line 904 is substantially a function that outputs 0 as the first zoom speed when the size difference is smaller than size difference 902, and outputs the first zoom speed calculated according to the above formula (3) when the size difference is equal to or greater than size difference 902. Furthermore, straight line 905 is substantially a function that outputs 0 as the second zoom speed when the size difference is smaller than size difference 901, and outputs the second zoom speed calculated according to the above formula (4) when the size difference is equal to or greater than size difference 901 and smaller than size difference 902.

[0160] Note that there is no limitation on which of formula (3) and formula (4) is used for the method of calculating the zoom speed when the size difference is equal to or greater than the size difference 902, but this embodiment assumes that the zoom speed is calculated by formula (3). This is because the result of formula (3) is greater than the result of formula (4) for the zoom speed when the size difference is equal to or greater than the size difference 902, and therefore it is easy to maintain tracking even when tracking a subject at high speed.

[0161] Therefore, in Figure 9B In the example shown, if the size difference is less than size difference 901, the zoom speed is 0, and if the size difference is equal to or greater than size difference 902, the zoom speed is the first zoom speed calculated by the above formula (3). If the size difference is "equal to or greater than size difference 901 and less than size difference 902", the method for obtaining the zoom speed is changed depending on whether the drive speed is equal to or higher than the fifth threshold or lower than the fifth threshold.

[0162] If the driving speed is equal to or higher than the fifth threshold, the zoom speed is the first zoom speed calculated by the above formula (3) (however, due to the dead zone processing, the zoom speed is 0). On the other hand, if the driving speed is lower than the fifth threshold, the zoom speed is calculated as follows.

[0163] If the size difference changes from the state of "size difference less than size difference 901" to the state of "size difference equal to or greater than size difference 901 and less than size difference 902", the zoom speed is the first zoom speed calculated by the above formula (3) (but through the processing of the dead zone, the zoom speed is 0).

[0164] If the size difference changes from the state of "size difference equal to or greater than size difference 902" to the state of "size difference equal to or greater than size difference 901 and less than size difference 902", the zoom speed is the second zoom speed calculated by the above formula (4).

[0165] Using the processing in steps S807 to S814, for example, at the start of the tracking operation, the tracking operation is slowly started at a low zoom speed obtained by formula (3). That is, tracking is started when the size difference exceeds the first dead zone. Next, during tracking, the zoom of the camera 100 is controlled to reduce the size difference, and as the size difference increases, a higher zoom speed given by formula (3) is obtained. Then, when the tracking operation is stopped after the size difference exceeds the third threshold, the control method is switched, and a higher zoom speed given by formula (4) is obtained to further reduce the size difference. Therefore, it is easy to make the subject size closer to the target size.

[0166] In addition, as a method for calculating the control method, this embodiment has described a method for obtaining the linear zoom speed given by Formula (3) or Formula (4), but the calculation method is not limited to this. For example, two types of quadratic curves having a relationship similar to Formula (3) and Formula (4) can be used to obtain the zoom speed. This allows the tracking operation to start slowly and allows for easier tracking operation even when the size difference increases during tracking.

[0167] In step S815, the CPU 101 reads the drive control module 303 from the storage device 103 to the RAM 102, stores it in the RAM 102, and executes the stored drive control module 303. The CPU 101 then derives drive parameters for performing a zoom operation at a desired zoom speed in the zoom direction from the control commands stored in the RAM 102. The drive parameters indicate parameters for driving and controlling a zoom motor (not shown) included in the drive unit 109. The CPU 101 then drives and controls the drive unit 109 via the drive I / F 108 based on the derived drive parameters. When the drive unit 109 is driven based on the drive parameters, the camera 100 changes the shooting magnification (zoom), that is, performs a zoom operation.

[0168] In step S816, similar to step S415 above, the CPU 101 determines whether the end condition for ending the tracking is satisfied. If the CPU 101 determines that the end condition is satisfied, it ends the tracking according to Figure 8A-1 and Figure 8A-2 On the other hand, if the CPU 101 determines that the end condition is not satisfied, it returns the process to step S801.

[0169] Next, we will refer to Figure 8B The flowchart shown is used to illustrate the operation of the controller 200.

[0170] In step S817, the CPU 201 reads the user interface module 306 from the storage device 203 to the RAM 202, stores it in the RAM 202, and executes the stored user interface module 306. The CPU 201 then displays the captured image received from the camera 100 and stored in the RAM 202 on the display unit 206. The CPU 201 executes the user interface module 306 stored in the RAM 202. The CPU 201 then accepts a touch operation for a “tracking subject target size setting operation” performed by the user on the display unit 206.

[0171] For example, the user can touch an option (displayed on the display unit 206) of a predetermined target size such as full body, upper body, or bust shot with his / her finger, thereby setting the target size corresponding to the touched option. Alternatively, the user can input a numerical value as the target size.

[0172] Note that the method for setting the target size is not limited to a specific method. For example, the user can set the target size by operating the user input I / F 207. Then, the CPU 201 determines whether a touch operation for "tracking subject target size setting operation" has been input.

[0173] If the CPU 201 determines that a touch operation for “setting the tracking subject target size” has been input, it stores the target size set by the touch operation in the RAM 202 and transfers the process to step S818. On the other hand, if the CPU 201 determines that a touch operation for “setting the tracking subject target size” has not been input, it returns the process to step S817.

[0174] In step S818, the CPU 201 reads the communication module 309 from the storage device 203 to the RAM 202, deploys it in the RAM 202, and executes the communication module 309. The CPU 201 then sends the new target size stored in the RAM 202 to the camera 100 via the network I / F 205.

[0175] As described above, by determining whether the tracking subject has moved based on the size difference between the target size and the subject size and switching the control operation, the subject size can be easily brought closer to the target size if the tracking subject has moved.

[0176] That is, in the present embodiment, in a structure for calculating a control speed for controlling the zoom of a camera device so that the detected size of a subject detected in an image captured by the camera device becomes closer to a target size of the subject in the captured image, the following camera device example has been described: in a state where a difference between the detected size and the target size is equal to or greater than a first threshold value, a first control speed is calculated as the control speed for the zoom based on the difference, and in a case where after the above state the difference becomes less than the first threshold value and equal to or greater than a second threshold value which is smaller than the first threshold value, a second control speed higher than the first control speed is calculated as the control speed for the zoom based on the difference.

[0177] Note that, in a case where the difference becomes smaller than the first threshold and equal to or larger than the second threshold after the state in which the difference is smaller than the second threshold, the first control speed may be calculated as the control speed of zooming based on the difference.

[0178] In this embodiment, the controller 200 can also perform some or all of the calculations for the drive amount used to detect and track the subject. In this case, the camera 100 first transmits a captured image to the controller 200. The controller 200 detects the tracking subject from the received captured image, calculates drive parameters for tracking the tracking subject (as described above), and transmits the drive parameters to the camera 100. The camera 100 then tracks the tracking subject based on the received drive parameters (as described above) and captures an image of the tracking subject.

[0179] The processing for calculating the control speed (angular rate or zoom speed) described in one or more of the above embodiments may be performed by the camera 100 or the controller 200, or may be performed by a device separate from these devices. In other words, the computing device that performs the processing for calculating the control speed (angular rate or zoom speed) may be incorporated into the camera 100 or the controller 200, or may exist as a device separate from these devices. The computing device may be implemented by hardware, software, or a combination thereof.

[0180] For example, it can be configured to stop the zoom of the camera 100 by judging the threshold value of the distance difference, and stop the pan / tilt of the camera 100 by judging the threshold value of the size difference. This can simultaneously operate the pan, tilt and zoom of the camera 100, thereby improving video quality.

[0181] The numerical values, processing timing, processing order, processing entity, data (information) structure / acquisition method / sending destination / sending source / storage location, etc. used in each of the above embodiments are merely examples for specific description and are not intended to limit the scope and features of the present disclosure to such examples.

[0182] Furthermore, a part or all of the above-described embodiments may be appropriately combined and used. A part or all of the above-described embodiments may be selectively used.

[0183] Other embodiments

[0184] The embodiments of the present invention can also be implemented by the following method, that is, software (including computer program products of computer programs / instructions) that perform the functions of the above-mentioned embodiments is supplied to a system or device via a network or various storage media, and a computer (central processing unit (CPU) or microprocessing unit (MPU)) of the system or device reads and executes the computer program / instructions.

[0185] Although one or more features of the present disclosure have been described with reference to exemplary embodiments, it should be understood that the scope of the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A computing device, comprising: a calculation unit configured or operative to calculate a control speed for controlling the posture of an imaging device so that a detected position of an object detected from an image captured by the imaging device becomes closer to a target position of the object in the captured image, In which, in a state where the difference between the detection position and the target position is not less than a first threshold value, the calculation unit calculates a first control speed as the control speed of the posture based on the difference, and in a case where the difference becomes less than the first threshold value and not less than a second threshold value which is smaller than the first threshold value after the state, the calculation unit calculates a second control speed higher than the first control speed as the control speed of the posture based on the difference.

2. The device according to claim 1, wherein The calculation unit sets the control speed of the gesture to 0 if the difference is smaller than the second threshold.

3. The device according to claim 1, wherein When the difference becomes smaller than the first threshold and not smaller than the second threshold after the state in which the difference is smaller than the second threshold, the calculation unit calculates the first control speed as the control speed of the gesture based on the difference.

4. The device according to claim 1, wherein In a case where the control speed of the posture of the camera device is less than a threshold value, the calculation unit calculates the first control speed as the control speed of the posture based on the difference while the difference is not less than the first threshold value, and after this state, in a case where the difference becomes less than the first threshold value and not less than the second threshold value, the calculation unit calculates the second control speed as the control speed of the posture based on the difference.

5. The device according to claim 1, wherein The calculation unit calculates the first control speed as the control speed of the gesture based on the difference when the control speed of the gesture of the imaging device is not less than a threshold value.

6. The device according to claim 1, wherein The calculation unit sets the control speed of the gesture to 0 in a case where the difference becomes smaller than the first threshold and not smaller than the second threshold after a state in which the difference is smaller than the second threshold.

7. The apparatus according to claim 1, further comprising: A control unit is configured or operated to control the gesture according to the control speed of the gesture calculated by the calculation unit.

8. The apparatus according to claim 1, further comprising: A transmitting unit is configured or operated to transmit the control speed of the gesture calculated by the calculating unit to the imaging device.

9. A computing device comprising: a calculation unit configured or operative to calculate a control speed for controlling zooming of an imaging device so that a detected size of an object detected from an image captured by the imaging device becomes closer to a target size of the object in the captured image, In which, in a state where the difference between the detection size and the target size is not less than a first threshold value, the calculation unit calculates a first control speed as the control speed of the zoom based on the difference, and in a case where after the state the difference becomes less than the first threshold value and not less than a second threshold value which is smaller than the first threshold value, the calculation unit calculates a second control speed higher than the first control speed as the control speed of the zoom based on the difference.

10. The device according to claim 9, wherein In a case where the control speed of the zoom of the camera device is less than a threshold value, the calculation unit calculates the first control speed as the control speed of the zoom based on the difference while the difference is not less than the first threshold value, and after this state, in a case where the difference becomes less than the first threshold value and not less than the second threshold value, the calculation unit calculates the second control speed as the control speed of the zoom based on the difference.

11. The device according to claim 9, wherein The calculation unit calculates the first control speed as the zoom control speed based on the difference when the zoom control speed of the imaging device is not less than a threshold value.

12. The device according to claim 9, wherein The calculation unit sets the zoom control speed to 0 if the difference is smaller than the second threshold.

13. The device according to claim 9, wherein The calculation unit sets the control speed of the zoom to 0 in a case where the difference becomes smaller than the first threshold and not smaller than the second threshold after a state in which the difference is smaller than the second threshold.

14. The apparatus according to claim 9, further comprising: A control unit is configured or operated to control the zoom according to the control speed of the zoom calculated by the calculation unit.

15. The apparatus according to claim 9, further comprising: A sending unit is configured to send the zoom control speed calculated by the calculating unit to the camera device.

16. The device according to claim 9, wherein The calculation unit calculates the first control speed as the control speed of the zoom based on the difference when the difference becomes smaller than the first threshold and not smaller than the second threshold after the difference is smaller than the second threshold.

17. A computing method, executed by a computing device, comprising: calculating a control speed for controlling the posture of an imaging device so that a detection position of an object detected from an image captured by the imaging device becomes closer to a target position of the object in the captured image, In the calculation, in a state where the difference between the detection position and the target position is not less than a first threshold value, a first control speed is calculated as the control speed of the posture based on the difference, and after the state, when the difference becomes less than the first threshold value and not less than a second threshold value which is smaller than the first threshold value, a second control speed higher than the first control speed is calculated as the control speed of the posture based on the difference.

18. A computing method, executed by a computing device, comprising: calculating a control speed for controlling zooming of an imaging device so that a detected size of an object detected from an image captured by the imaging device becomes closer to a target size of the object in the captured image, In the calculation, in a state where the difference between the detection size and the target size is not less than a first threshold value, a first control speed is calculated as the control speed of the zoom based on the difference, and after the state, when the difference becomes less than the first threshold value and not less than a second threshold value which is smaller than the first threshold value, a second control speed higher than the first control speed is calculated as the control speed of the zoom based on the difference.

19. A computer-readable storage medium storing a computer program for causing a computer to perform the following method, the method comprising: calculating a control speed for controlling the posture of an imaging device so that a detection position of an object detected from an image captured by the imaging device becomes closer to a target position of the object in the captured image, In the calculation, in a state where the difference between the detection position and the target position is not less than a first threshold value, a first control speed is calculated as the control speed of the posture based on the difference, and after the state, when the difference becomes less than the first threshold value and not less than a second threshold value which is smaller than the first threshold value, a second control speed higher than the first control speed is calculated as the control speed of the posture based on the difference.

20. A computer-readable storage medium storing a computer program for causing a computer to perform the following method, the method comprising: calculating a control speed for controlling zooming of an imaging device so that a detected size of an object detected from an image captured by the imaging device becomes closer to a target size of the object in the captured image, In the calculation, in a state where the difference between the detection size and the target size is not less than a first threshold value, a first control speed is calculated as the control speed of the zoom based on the difference, and after the state, when the difference becomes less than the first threshold value and not less than a second threshold value which is smaller than the first threshold value, a second control speed higher than the first control speed is calculated as the control speed of the zoom based on the difference.

21. A computer program product storing a computer program for causing a computer to perform the following method, the method comprising: calculating a control speed for controlling the posture of an imaging device so that a detection position of an object detected from an image captured by the imaging device becomes closer to a target position of the object in the captured image, In the calculation, in a state where the difference between the detection position and the target position is not less than a first threshold value, a first control speed is calculated as the control speed of the posture based on the difference, and after the state, when the difference becomes less than the first threshold value and not less than a second threshold value which is smaller than the first threshold value, a second control speed higher than the first control speed is calculated as the control speed of the posture based on the difference.

22. A computer program product storing a computer program for causing a computer to perform the following method, the method comprising: calculating a control speed for controlling zooming of an imaging device so that a detected size of an object detected from an image captured by the imaging device becomes closer to a target size of the object in the captured image, In the calculation, in a state where the difference between the detection size and the target size is not less than a first threshold value, a first control speed is calculated as the control speed of the zoom based on the difference, and after the state, when the difference becomes less than the first threshold value and not less than a second threshold value which is smaller than the first threshold value, a second control speed higher than the first control speed is calculated as the control speed of the zoom based on the difference.

Images