Imaging device, imaging control program, and imaging control method

Abstract

To accurately recognize an object to accurately track the object even when a mode for moving an angle of view is changed.SOLUTION: In a tracking system, an imaging device 106 includes: a tracking operation control unit for switching a mode for moving an angle of view according to a scene; a tracking operation control unit for detecting and tracking an object present within the angle of view based on at least one image captured by the imaging device; and an imaging unit position control unit for converting a result of tracking the object into an instruction regarding the moving speed of the angle of view by using information indicating a correspondence between an instructed moving speed of the angle of view and a moving speed of the angle of view when actually moved. The tracking operation control unit also acquires angle-of-view position data indicating a position of the angle of view from the imaging device, compares the position of the angle of view indicated by the angle-of-view position data with the position of the angle of view predicted based on the instruction, and executes processing related to information correction processing for correcting information so that a difference exceeding a predetermined threshold is equal to or less than the predetermined threshold when there is the difference.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an imaging device, an imaging control program, and an imaging control method. [Background technology]

[0002] In surveillance and monitoring systems, pan-tilt-zoom (PTZ) cameras are frequently used, which are capable of panning to move the angle of view horizontally, tilting to move the angle of view vertically, and zooming to change the angle of view to telephoto or wide-angle. In addition, a technique for controlling the panning, tilting, and zooming of a PTZ camera in order to continuously capture an image of a specific object is known as a technique related to the PTZ camera.

[0003] When tracking an object using the above-mentioned technology, the performance of tracking the object depends on the accuracy of object recognition that recognizes the object in the image and the performance of control that moves the PTZ. For example, in order to accurately track an object using the above-mentioned technology, it is necessary to continue to accurately recognize the object in the image and to continue to move the PTZ at an appropriate speed accordingly. Object recognition has been improved by improving software algorithms, using artificial intelligence, etc. For example, Patent Document 1 discloses a technology that accurately tracks an object even when multiple recognized objects cross each other.

[0004] On the other hand, the performance of tracking an object differs depending on the type of PTZ camera. Therefore, it is necessary to analyze the performance of moving the PTZ for each PTZ camera and change the settings of the PTZ camera so that the PTZ moves according to the performance. Therefore, when a tracking system including a PTZ camera is started for the first time, it often executes a process of measuring the PTZ movement speed when the PTZ is actually moved, and generating and storing a speed table associated with the movement speed of the specified angle of view. This process is also called calibration.

[0005] Also, PTZ cameras are sometimes used to capture images of moving objects by remote control. The speed of the object varies greatly depending on the scene. For example, in a scene where a sport is being played, the tracking system needs to quickly track the ball, players, etc. Also, in a scene such as a stage or a class, the tracking system needs to move the angle of view of the PTZ camera relatively slowly and minutely. As such, since the tracking system needs to move the angle of view of the PTZ camera to suit various scenes, some tracking systems are capable of switching the mode for moving the angle of view. In such a tracking system, even if the same operation is performed, the behavior of the angle of view can be switched by switching the mode for moving the angle of view. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-96262 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the above-mentioned tracking system, when the mode for moving the angle of view is changed after the calibration is performed, the instruction regarding the speed at which the angle of view is moved and the performance for actually moving the angle of view may be inconsistent, which may cause the above-mentioned tracking system to be unable to accurately recognize an object or to accurately track an object.

[0008] In view of these problems, the present invention aims to provide an imaging device, an imaging control program, and an imaging control method that are capable of accurately performing object recognition and accurately tracking objects even when the mode for moving the angle of view is changed. [Means for solving the problem]

[0009] In order to solve the above-mentioned problems, the imaging device of the present invention comprises a switching means for switching a mode for moving the angle of view of the imaging device in accordance with a scene in which the imaging device is used; a tracking means for detecting and tracking an object present within the angle of view based on at least one image captured by the imaging device; a speed table holding means for holding a speed table indicating a correspondence between a specified movement speed of the angle of view and a movement speed of the angle of view when actually moved; an instruction generating means for converting a result of tracking the object by the tracking means into an instruction regarding the movement speed of the angle of view using the speed table; and a speed table correction means for acquiring angle of view position data indicating a position of the angle of view from the imaging device, comparing the position of the angle of view indicated by the angle of view position data with the position of the angle of view predicted based on the instruction, and, if a difference exceeding a predetermined threshold occurs, executing processing related to a speed table correction process to correct the speed table so that the difference is equal to or less than the predetermined threshold. Effect of the Invention

[0010] According to the present invention, it is possible to provide an imaging device, an imaging control program, and an imaging control method that are capable of accurately performing object recognition and accurately tracking an object even when the mode for moving the angle of view is changed. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a tracking system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of a control unit according to an embodiment of the present invention. [Diagram 3] 2 is a diagram illustrating an example of a hardware configuration of a control unit excluding a communication I / F unit in an embodiment of the present invention. FIG. [Figure 4] 10 is a schematic diagram showing the relationship between the movement speed of the instructed angle of view and the speed of the PTZ motor, which is used when adjusting the motor control command in the embodiment of the present invention; FIG. [Diagram 5]10 is a flowchart illustrating an example of a calibration for generating a PT speed table according to an embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating an example of a tracking process including an operation of automatically detecting a change in a PTZ operation mode and correcting a PT speed table according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The imaging device according to the embodiment includes a switching means, a tracking means, a speed table holding means (information holding means), an instruction generating means, and a speed table correcting means (information correcting means). The switching means switches a mode for moving the angle of view of the imaging device according to a scene in which the imaging device is used. The tracking means detects and tracks an object present within the angle of view based on at least one image captured by the imaging device. The speed table holding means (information holding means) holds a speed table (correspondence information) indicating a correspondence between a movement speed of the instructed angle of view and a movement speed of the angle of view when actually moved. Note that this correspondence information does not necessarily need to be held in a table format, and any format may be used as long as it indicates a correspondence between a movement speed of the instructed angle of view and a movement speed of the angle of view when actually moved. The instruction generating means converts a result of tracking an object by the tracking means into an instruction regarding the movement speed of the angle of view using the speed table. The speed table correction means acquires angle-of-view position data indicating the position of the angle of view from the imaging device, and compares the position of the angle of view indicated by the angle-of-view position data with the position of the angle of view predicted based on the instruction. If a difference exceeding a predetermined threshold occurs between these two positions, the speed table correction means (information correction means) executes processing related to a speed table correction process (information correction process) that corrects the speed table so that the difference becomes equal to or smaller than the predetermined threshold.

[0013] Also, the speed table correction means (information correction means) may automatically execute the speed table correction process (information correction process) as a process related to the speed table correction process (information correction process). Furthermore, in this case, the speed table correction means (information correction process) may execute the speed table correction process (information correction process) at a time designated in advance.

[0014] Furthermore, the speed table correcting means (information correcting means) may execute a process for outputting a notification prompting the execution of the speed table correcting process (information correcting process) as a process related to the speed table correcting process (information correcting process).

[0015] Furthermore, the speed table correcting means (information correcting means) may execute a speed table correcting process (information correcting process) when the mode is switched by the switching means.

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a diagram showing an example of the configuration of a tracking system according to an embodiment. As shown in Fig. 1, the tracking system includes a PTZ camera 106, a network 104, a display device 101, an input device 103, a recording device 102, and an authentication information recording device 107.

[0017] The network 104 is, for example, the Internet, a mobile phone network, a LAN (Local Area Network), or a WAN (Wide Area Network). The network 104 also connects the PTZ camera 106, the display device 101, the recording device 102, and the authentication information recording device 107 in a manner that allows them to communicate with each other.

[0018] The input device 103 is used by a user to input data indicating instructions to be given to the PTZ camera 106. The input device 103 is also connected to the display device 101 in a communicable manner.

[0019] In response to a request from the display device 101, the PTZ camera 106 transmits video to the display device 101 via the network 104. In addition, in response to a request from the recording device 102, the PTZ camera 106 transmits video to the recording device 102 via the network 104. The recording device 102 stores the video obtained from the PTZ camera 106 in a recording medium such as a hard disk drive (HDD), a solid state drive (SDD), or an optical disk.

[0020] The display device 101 reads and displays images obtained from the PTZ camera 106, images stored in the recording device 102, etc. The display device 101 also receives requests from a user through the input device 103 and transmits the requests to the PTZ camera 106. The authentication information recording device 107 holds information for identifying an object.

[0021] Fig. 2 is a diagram showing an example of a functional configuration of a control unit according to an embodiment of the present invention. As shown in Fig. 2, the PTZ camera 106 includes an imaging unit 200 and a control unit 250. In addition, the PTZ camera 106 can change the imaging range by changing the PTZ.

[0022] The imaging unit 200 includes an imaging element, an optical system, and a camera platform. The imaging element is, for example, a solid-state imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) sensor or a CCD (Charge-Coupled Device) sensor. The optical system includes a lens and the like for forming an image of light incident from outside the PTZ camera 106 on the imaging element. The camera platform can support the entire housing in which the imaging element, the optical system, and the like are housed in any position.

[0023] As shown in FIG. 2, the controlled object 250 includes a communication I / F unit 251, a video input unit 252, a video encoding unit 253, a video transmission unit 254, an input unit 255, a tracking control unit 240, and an imaging unit position control unit 263.

[0024] The communication I / F unit 251 is responsible for sending and receiving information based on a communication method such as HTTP (Hypertext Transfer Protocol) or TCP / IP (Transmission Control Protocol / Internet Protocol), and functions as an interface with the network 104.

[0025] The video input unit 252 receives imaging data from the imaging unit 200. The imaging data includes video captured by the imaging unit 200 and data indicating the attitude of the imaging unit 200 when the video was captured. The attitude of the imaging unit 200 here refers to the pan value, tilt value, and zoom value of the imaging unit 200.

[0026] The video encoding unit 253 receives the captured image data from the video input unit 252, and encodes the video using a predetermined encoding method, such as JPEG, MPEG-2, MPEG-4, H.264, and H.265.

[0027] The video transmission unit 254 transmits the video encoded by the video encoding unit 253 to the outside via the communication I / F unit 251.

[0028] The input unit 255 receives, via the communication I / F unit 251, user input acquired via the input device 103, data selected based on the user input, and the like.

[0029] The tracking control unit 240 controls the tracking of an object by performing operations from object detection to control of an imaging position. As shown in FIG. 2, the tracking control unit 240 includes an object detection unit 256, an object tracking unit 257, a registration unit 258, a trajectory recording unit 259, a position prediction unit 260, a determination unit 261, and a tracking operation control unit 262.

[0030] The object detection unit 256 receives a captured image from the imaging data acquired by the video input unit 252. Then, the object detection unit 256 uses pattern recognition, machine learning, and the like to detect objects such as faces and human bodies from the captured image.

[0031] The object tracking unit 257 executes a process of matching corresponding objects between a plurality of images acquired in a time series. That is, the object tracking unit 257 executes a process of matching corresponding objects between temporally consecutive images. This process is a process of solving a matching problem formulated as a problem of pairing an object detected in one image with an object detected in another image.

[0032] First, the object tracking unit 257 defines the similarity of objects using values ​​such as the position and size of a figure representing the object and feature values ​​extracted from the object. Next, the object tracking unit 257 can solve the matching problem using a method of pairing objects in order starting from the pair with the highest similarity, a global optimization method of pairing multiple objects so that the sum of the similarities between them is maximized, or the like. Then, the object tracking unit 257 assigns the same ID to the associated objects.

[0033] The determination unit 261 determines whether or not the object being tracked by the object tracking unit 257 is a target specified by the user, based on the user input and data obtained from the input unit 255. Then, the determination unit 261 sets the object determined to be a target specified by the user as an object to be tracked.

[0034] The determination unit 261 uses, for example, object recognition as a method for determining whether an object is a specified target. Object recognition is a technique for determining the type of an object by checking against the characteristics of the object registered in advance. The determination unit 261 acquires data indicating the characteristics of the object that is registered in advance, for example, from the authentication information recording device 107. Note that the determination unit 261 may determine whether an object is a specified target by a method other than object recognition. For example, the determination unit 261 may determine whether an object is a specified target based on data input using the input device 103.

[0035] The registration unit 258 registers data indicating the results of the processing executed by the object tracking unit 257 in a holding unit (not shown).

[0036] The trajectory recording unit 259 records the trajectory of an object that is a candidate for tracking processing, based on data registered in a storage unit (not shown).

[0037] When an object that is a candidate for tracking processing does not exist within the range captured by the imaging unit 200, the position prediction unit 260 predicts the current position of the object based on the information recorded by the tracking record unit 259.

[0038] The tracking operation control unit 262 is an example of the tracking means described above. The tracking operation control unit 262 determines the position for capturing an image so that the object is within the range captured by the imaging unit 200, and instructs the imaging unit position control unit 263 to move the angle of view to the determined position. The relationship between the content of the instruction to the imaging unit position control unit 263 and the actual change in the position of the angle of view of the imaging unit 200 has characteristics for each type of PTZ camera 106. Therefore, when the tracking system operates for the first time, it issues multiple instructions to the imaging unit position control unit 263 and acquires data indicating the attitude of the imaging unit 200 from the video input unit 252, and holds the correspondence internally as a PT speed table (correspondence information). The instruction to the imaging unit position control unit 263 is generated taking the PT speed table into consideration.

[0039] The imaging unit position control unit 263 receives an instruction to move the angle of view of the imaging unit 200, which is input by a user and acquired from the tracking operation control unit 262 or the input unit 255. The imaging unit position control unit 263 generates motor control commands for controlling the pan motor, tilt motor, and zoom motor of the PTZ camera 106 in accordance with the received instruction, and transmits the motor control commands to the imaging unit 200. Note that the imaging unit position control unit 263 is an example of the instruction generating means described above. Furthermore, when generating the command, the imaging unit position control unit 263 adjusts the motor control command by referring to the operation mode of the PTZ.

[0040] Fig. 4 is a schematic diagram showing the relationship between the instructed moving speed of the angle of view and the speed of the PTZ motor, which is used when adjusting the motor control command according to the embodiment. Fig. 4 shows that the speed of the PTZ motor is different between high speed and low speed, which are PTZ operation modes.

[0041] Next, an example of calibration for generating a PT speed table will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of calibration for generating a PT speed table according to an embodiment of the present invention. When the tracking system is started, the tracking operation control unit 262 checks whether a PT speed table is held, and if it is confirmed that the table is not held, executes the calibration described below.

[0042] In step S401, the tracking operation control section 262 specifies to the imaging section position control section 263 the position of the predetermined angle of view of the imaging section 200 and the moving speed of the predetermined angle of view of the imaging section 200, and causes the PT to operate.

[0043] In step S 402 , the tracking operation control section 262 acquires data indicating the attitude of the imaging section 200 , that is, data indicating the pan value, tilt value, and zoom value of the imaging section 200 from the video input section 252 .

[0044] In step S403, the tracking operation control unit 262 measures the moving speed of the actual PT of the angle of view of the imaging object 200, and associates the speed of the instructed angle of view with the actual moving speed of the angle of view.

[0045] The tracking operation control unit 262 changes the speed of the instructed angle of view and then executes the processes from step S401 to step S403 multiple times.

[0046] In step S404, the tracking operation control unit 262 creates a PT speed table based on the results of performing the processes from step S401 to step S403 multiple times. The PT speed table is stored, for example, by the speed table storage means described above. The speed table storage means is a recording unit (not shown), and can be realized, for example, by a recording medium such as an HDD or SSD.

[0047] Next, a tracking process including an operation of automatically detecting a change in a PTZ operation mode and correcting a PT speed table will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of a tracking process including an operation of automatically detecting a change in a PTZ operation mode and correcting a PT speed table according to an embodiment of the present invention.

[0048] In step S 501 , the object detection unit 256 acquires a captured image from the imaging data acquired by the video input unit 252 .

[0049] In step S502, the tracking operation control section 262 acquires data indicating the attitude of the imaging section 200 from the imaging data acquired by the video input section 252, that is, data indicating the pan value, tilt value, and zoom value of the imaging section 200.

[0050] In step S503, the tracking operation control unit 262 determines whether the difference between the position of the angle of view stored in the PT predicted position table and the position of the angle of view indicated by the data acquired in step S502 is equal to or greater than a predetermined value. If the tracking operation control unit 262 determines that the difference is equal to or greater than the predetermined value (step S503: YES), the process proceeds to step S504. On the other hand, if the tracking operation control unit 262 determines that the difference is less than the predetermined value (step S503: NO), the process proceeds to step S505.

[0051] In step S504, the tracking operation control section 262 corrects the PT speed table. That is, the tracking operation control section 262 is one example of the above-mentioned speed table correction means (information correction means).

[0052] In step S505, the object detection unit 256 executes a process of detecting an object such as a face or a human body from a given captured image by using various techniques such as known pattern recognition and machine learning.

[0053] In step S506, the object detection unit 256 determines whether or not an object was detected in step S505. If the object detection unit 256 determines that an object was detected in step S505 (step S505: YES), the process proceeds to step S507. On the other hand, if the object detection unit 256 determines that an object was not detected in step S505 (step S505: NO), the process ends.

[0054] In step S507, the object tracking unit 257 and the determination unit 261 determine whether or not the object detected in step S505 is an object to be tracked. Then, in step S507, the position prediction unit 260 determines the target position of the PTZ.

[0055] In step S508, the tracking operation control unit 262 creates a PTZ control instruction using the PT speed table to move the PTZ to the target position determined in step S507, and transmits the PTZ control instruction to the imaging unit position control unit 263.

[0056] In step S509, the tracking operation control unit 262 creates a PT position prediction table that lists, in chronological order, the PT positions of the imaging unit 200 that are expected after a PTZ movement instruction is executed.

[0057] In step S510, the tracking operation control unit 262 adjusts and generates a motor control command from the received PTZ control instruction using the relationship shown in FIG.

[0058] The processes from step S501 to step S510 are repeated until the object to be tracked is no longer detected in step S506.

[0059] The present invention includes the following inventions that appropriately combine the above-mentioned features.

[0060] (Configuration 1) an instruction generating means for converting a result of tracking of the object by the tracking means into an instruction regarding the speed of movement of the angle of view, using information indicating a correspondence between a specified speed of movement of the angle of view and the speed of movement of the angle of view when actually moved; and a speed table correction means for acquiring angle of view position data indicating a position of the angle of view from the imaging device, comparing the position of the angle of view indicated by the angle of view position data with the position of the angle of view predicted based on the instruction, and executing processing related to an information correction process for correcting the information so that the difference is equal to or less than the specified threshold if a difference exceeding a predetermined threshold occurs.

[0061] (Configuration 2) 2. The imaging device according to configuration 1, wherein the information correction means automatically executes the information correction process.

[0062] (Configuration 3) 3. The imaging device according to configuration 2, wherein the information correction means executes the information correction process at a time designated in advance.

[0063] (Configuration 4) 2. The imaging device according to configuration 1, wherein the information correction means outputs a notification prompting the user to execute the information correction process.

[0064] (Configuration 5) 5. The imaging device according to any one of configurations 1 to 4, wherein the information correction means executes the information correction process when the mode is switched by the switching means.

[0065] (Program 1) an imaging control program that causes a computer to execute a process of: switching a mode for moving an angle of view of an imaging device in accordance with a scene in which the imaging device is used; detecting and tracking an object present within the angle of view based on at least one image captured by the imaging device; converting a result of tracking the object by the tracking means into an instruction regarding the speed of movement of the angle of view using information indicating a correspondence between a specified speed of movement of the angle of view and a speed of movement of the angle of view when actually moved; acquiring angle of view position data that indicates a position of the angle of view from the imaging device; comparing the position of the angle of view indicated by the angle of view position data with the position of the angle of view predicted based on the instruction; and, if a difference exceeding a predetermined threshold occurs, executing a process related to an information correction process that corrects the information so that the difference is equal to or less than the predetermined threshold.

[0066] (Method 1) an imaging control method comprising: switching a mode for moving an angle of view of an imaging device in accordance with a situation in which the imaging device is used; detecting and tracking an object present within the angle of view based on at least one image captured by the imaging device; converting a result of tracking the object by the tracking means into an instruction regarding the speed of movement of the angle of view using information indicating a correspondence between a specified speed of movement of the angle of view and the speed of movement of the angle of view when actually moved; acquiring angle of view position data indicating a position of the angle of view from the imaging device; comparing the position of the angle of view indicated by the angle of view position data with the position of the angle of view predicted based on the instruction; and, if a difference exceeding a predetermined threshold occurs, executing processing related to an information correction process that corrects the information so that the difference is equal to or less than the predetermined threshold.

[0067] The above describes preferred embodiments of the present invention. However, the present invention is not limited to the above-described embodiments. In other words, the present invention includes embodiments in which various modifications have been made based on the gist of the present invention, and these embodiments are not excluded from the scope of the present invention. [Explanation of symbols]

[0068] 106:PTZ camera 200: Imaging unit 240: Tracking control unit 250: Control unit

Claims

1. a switching means for switching a mode for moving the angle of view of the imaging device in accordance with a situation in which the imaging device is used; a tracking means for detecting and tracking an object present within the angle of view based on at least one image captured by the imaging device; an instruction generating means for converting a result of tracking the object by the tracking means into an instruction regarding the moving speed of the angle of view, using information indicating a correspondence relationship between a specified moving speed of the angle of view and a moving speed of the angle of view when the object actually moves; an information correcting means for acquiring angle-of-view position data indicating a position of the angle of view from the imaging device, comparing the position of the angle of view indicated by the angle-of-view position data with a position of the angle of view predicted based on the instruction, and, if a difference exceeding a predetermined threshold occurs, executing a process related to an information correcting process for correcting the information so that the difference becomes equal to or smaller than the predetermined threshold; An imaging device comprising:

2. The information correction means automatically executes the information correction process.

2. The imaging device according to claim 1 .

3. The information correcting means executes the information correcting process at a pre-specified time.

3. The imaging device according to claim 2.

4. The information correction means outputs a notification prompting the user to execute the information correction process.

2. The imaging device according to claim 1 .

5. the information correcting means executes the information correcting process when the mode is switched by the switching means.

3. The imaging device according to claim 1 or 2.

6. On the computer, switching a mode for shifting the angle of view of the imaging device in accordance with a situation in which the imaging device is used; Detecting and tracking an object present within the angle of view based on at least one image captured by the imaging device; converting a result of tracking the object by the tracking means into an instruction regarding the moving speed of the angle of view by using information indicating a correspondence relationship between the instructed moving speed of the angle of view and the moving speed of the angle of view when actually moved; acquiring, from the imaging device, angle-of-view position data indicating a position of the angle of view, comparing the position of the angle of view indicated by the angle-of-view position data with the position of the angle of view predicted based on the instruction, and, if a difference exceeding a predetermined threshold occurs, executing a process related to an information correction process for correcting the information so that the difference becomes equal to or smaller than the predetermined threshold; An imaging control program for causing a processing to be executed.

7. switching a mode for shifting the angle of view of the imaging device in accordance with a situation in which the imaging device is used; Detecting and tracking an object present within the angle of view based on at least one image captured by the imaging device; converting a result of tracking the object by the tracking means into an instruction regarding the moving speed of the angle of view by using information indicating a correspondence relationship between the instructed moving speed of the angle of view and the moving speed of the angle of view when actually moved; acquiring, from the imaging device, angle-of-view position data indicating a position of the angle of view, comparing the position of the angle of view indicated by the angle-of-view position data with the position of the angle of view predicted based on the instruction, and, if a difference exceeding a predetermined threshold occurs, executing a process related to an information correction process for correcting the information so that the difference becomes equal to or smaller than the predetermined threshold; 4. An imaging control method comprising:

Images