Control unit, imaging apparatus, control method, and program

The control device adjusts imaging device speeds based on detection results to maintain objects or scenes of interest, addressing the challenge of rapid range changes and improving operator understanding during patrol operations.

JP2024165552APending Publication Date: 2024-11-28CANON KK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023081832
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing imaging systems with pan/tilt functions face challenges in maintaining operator understanding during patrol operations due to rapid changes in imaging range, which can cause objects of interest to move out of view or distract operators with multiple targets.

Method used

A control device and method that adjusts the operating speed of imaging devices based on detection results, such as the number of people or scene brightness, to extend the viewing time of relevant content during patrol operations.

Benefits of technology

Enhances operator understanding by maintaining objects or scenes of interest within the view for a longer duration, reducing the complexity of monitoring multiple targets and improving video evaluation efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024165552000001_ABST
    Figure 2024165552000001_ABST
Patent Text Reader

Abstract

To output images allowing an operator to easily grasp the details of the images during a cyclic operation.SOLUTION: A control unit controls an imaging apparatus provided with driving means that performs the operation of at least one of panning, tilting, and zooming, and the control unit has: acquisition means that acquires images picked up by imaging means of the imaging apparatus; detection means that detects an object included in the images acquired by the acquisition means; and control means that controls the operation of the driving means. In a process of causing the driving means to perform a predetermined series of operations, the control means changes the operating speed of the driving means on the basis of a result of detection made by the detection means.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a control device, an imaging device, a control method, and a program, and more particularly to drive control of an imaging device that continuously switches an imaging range. [Background technology]

[0002] There is a system that assists in monitoring a target area by sequentially switching and displaying surveillance images from multiple imaging devices installed with different imaging ranges. Patent Document 1 discloses a control technology that, in order to assist an operator (surveillant) in understanding the contents of such a system, extends the display time of surveillance images in which a large number of objects are detected, making it difficult to switch to surveillance images from another imaging device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-125974 A Summary of the Invention [Problem to be solved by the invention]

[0004] By the way, the above-mentioned system employs an imaging device equipped with a pan / tilt function, which can expand the imaging range covered by one imaging device, and can reduce the types of surveillance images that an operator must monitor, for example. Also, by making such an imaging device perform a predetermined pan / tilt operation to output surveillance images whose imaging range changes sequentially, a so-called patrol operation, the burden of operating the imaging device on the operator can be reduced.

[0005] However, when the imaging range captured in the surveillance video changes sequentially due to the patrol operation, it may be difficult for the operator to understand the contents. That is, even if an object of interest is captured in the surveillance video, if the imaging range changes due to the patrol operation, the object may go out of the field of view, or the operator may focus on another object.

[0006] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a control device, an imaging device, a control method, and a program that output images whose contents are easy to understand during patrol operation. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the control device of the present invention is a control device that controls an imaging device equipped with a driving means that performs at least one of pan, tilt, and zoom operations, and has an acquisition means that acquires an image captured by the imaging means of the imaging device, a detection means that detects an object included in the image acquired by the acquisition means, and a control means that controls the operation of the driving means, and is characterized in that the control means changes the operating speed of the driving means based on the detection results by the detection means in the process of causing the driving means to perform a predetermined series of operations. Effect of the Invention

[0008] With this configuration, the present invention makes it possible to output images whose contents are easy to understand during a patrol operation. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a configuration of an imaging system according to an embodiment and a modification of the present invention. [Diagram 2] FIG. 1 is a block diagram illustrating a functional configuration of an image capture device 100 according to an embodiment and a modification of the present invention. [Diagram 3] FIG. 1 is a diagram illustrating a user interface for specifying deceleration conditions according to the first embodiment of the present invention; [Figure 4] 1 is a flowchart illustrating an example of an operation control process executed in the image capture device 100 according to the first embodiment of the present invention. [Diagram 5] FIG. 1 is a diagram illustrating the operation control of a patrol operation according to the first embodiment of the present invention; [Figure 6] FIG. 11 is a diagram illustrating a user interface for specifying a deceleration condition according to a second embodiment of the present invention; [Figure 7] 11 is a flowchart illustrating an operation control process executed in the image capture device 100 according to the second embodiment of the present invention. [Figure 8] FIG. 11 is a diagram illustrating the operation control of the patrol operation according to the second embodiment of the present invention; [Figure 9] FIG. 11 is a diagram illustrating a user interface for specifying a deceleration condition according to a third embodiment of the present invention. [Figure 10] 11 is a flowchart illustrating an operation control process executed in the image capture device 100 according to the third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] [Embodiment 1] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] In the embodiment described below, an example of the present invention is applied to an imaging device configured to be capable of panning, tilting, and zooming, as an example of an imaging device. However, the present invention can be applied to any device capable of controlling an imaging device having a driving unit that performs at least one of panning, tilting, and zooming.

[0012] Imaging system configuration FIG. 1 illustrates an example of the configuration of an imaging system according to this embodiment. As illustrated, the imaging system of this embodiment is composed of an imaging device 100 configured to be capable of panning, tilting, and zooming, and a user terminal 200 that receives and displays an image captured by the imaging device 100. The imaging device 100 and the user terminal 200 are configured to be connectable via a network 300, which may be, for example, a LAN or a WAN. The user terminal 200 may be any device having a display function, an operation input function, and a communication function. The user terminal 200 not only receives an image from the imaging device 100 by using the communication function, but also transmits information based on an operation input made at the user terminal 200 to the imaging device 100. In the example of FIG. 1, only one imaging device 100 is included in the imaging system, and the user terminal 200 communicates with the imaging device 100, but the user terminal 200 may be configured to be able to communicate with multiple imaging devices 100.

[0013] Configuration of the Imaging Device 100 Next, the functional configuration of the imaging device 100 according to this embodiment will be described with reference to the block diagram of FIG.

[0014] The imaging unit 1 includes an imaging optical system and an imaging element. The imaging optical system includes a plurality of lenses, an aperture, an optical filter such as an infrared cut filter, etc. The imaging element is a photoelectric conversion element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 1 outputs an image signal (hereinafter, may be simply referred to as an image) obtained by the imaging element performing photoelectric conversion on an optical image formed on an imaging surface via the imaging optical system. Here, the image signal will be described as a digital image signal to which A / D conversion processing has been applied, but it may also be an analog image signal. In this case, the A / D conversion processing is applied in the image processing unit 3 at the subsequent stage.

[0015] The driving unit 2 is a driving device that changes the imaging direction or imaging angle of view of the imaging unit 1. As described above, the imaging device 100 of this embodiment is configured to be capable of panning, tilting, and zooming, and the driving unit 2 includes a motor corresponding to each operation and a drive control circuit for the motor. The state of the motor is acquired by an encoder (not shown), and the driving unit 2 outputs information on the current pan angle, tilt angle, and zoom magnification of the imaging unit 1 to the control unit 8 based on the output value of the encoder.

[0016] The image processing unit 3 performs various image processing on the image output from the imaging unit 1. The image processing unit 3 performs, for example, color balance processing, gamma correction processing, etc. on the output image. The image data obtained by applying the image processing is output to the memory 4 and the detection unit 6. The output image data may be converted into an image file in a predetermined format such as JPEG (Joint Photographic Experts Group), H.264, H.265, etc., or a video file.

[0017] The memory 4 is a storage device included in the imaging device 100. The memory 4 may include a volatile memory such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), and a non-volatile memory such as a flash memory. The memory 4 not only stores the video data output from the image processing unit 3, but also stores various information necessary for the operation of the imaging device 100. The various information necessary for the operation of the imaging device 100 includes information for detecting an image of a person included in a video in a recognition process executed by a detection unit 6 described later. In addition, the memory 4 also includes an operation program for each block.

[0018] The communication unit 5 is a communication I / F with an external device provided in the imaging device 100. The communication unit 5 transmits the video data stored in the memory 4 to the user terminal 200 via the network 300. The communication unit 5 also transmits to the user terminal 200 information on a graphical user interface (GUI) required for various settings in order to accept various settings for operation control from the user terminal 200. The communication unit 5 can also receive information related to various settings from the user terminal 200 using the GUI and store it in the memory 4.

[0019] The detection unit 6 detects human images from the video data output from the image processing unit 3. In this embodiment, the detection unit 6 obtains feature information for detecting human images from the memory 4, and outputs to the control unit 8 the presence or absence of human images in the video data and the number of detected images based on the information.

[0020] The operation control unit 7 controls the driving unit 2 to perform a predetermined patrol operation when the imaging device 100 is set to a state in which the patrol operation is performed. The state in which the patrol operation is performed may be set, for example, when a request is received from the user terminal 200 via the network 300, or may be set by the control unit 8 when a predetermined condition is satisfied. The operation control unit 7 sequentially supplies control information including at least one of a pan angle, a tilt angle, and a zoom magnification to be set to the driving unit 2 according to, for example, information on a predetermined patrol operation. In this embodiment, the patrol operation is set so that the pan angle, tilt angle, and zoom magnification are the same at the start and end of a series of operations, and the patrol operation is controlled to be repeatedly performed during the period in which the imaging device 100 is set to a state in which the patrol operation is performed.

[0021] The control unit 8 controls the operation of each block of the imaging device 100. Specifically, the control unit 8 reads out an operation program for each block stored in the nonvolatile memory of the memory 4, expands it in the volatile memory, and executes it to control the operation of each block.

[0022] In this embodiment, various processes are described as being realized by circuits and processors corresponding to the respective blocks of the hardware of the imaging device 100. However, the embodiment of the present invention is not limited to this, and the process of each block may be realized by a program that performs the same process as that of each block.

[0023] <<Overview of Motion Control>> Next, an overview of the operation control of the imaging device 100 using the imaging system of this embodiment will be described. In the use case described below, the imaging system is used for monitoring purposes, and the image captured by the imaging device 100 is displayed on the user terminal 200 used by the operator. In this use, the imaging device 100 sequentially captures images while performing a predetermined patrol operation, and sequentially transmits the obtained images to the user terminal 200. The operator can grasp the behavior of the person captured in the image via the image displayed on the user terminal 200. Although the description is omitted in this specification, the operator may be able to remotely monitor the state of the area where the imaging device 100 is installed via the image, and, if necessary, obtain the operation authority of the imaging device 100 to change it to a desired state (pan, tilt, and zoom) to capture the image.

[0024] However, in surveillance applications, the image presented to the operator often requires the operator's attention. For example, when a person appears in the imaging range, the operator needs to watch the image captured in the imaging range and determine whether the person is behaving suspiciously. When the imaging device 100 performs a patrol operation, the imaging range changes sequentially, so the operator needs to make such a determination while the person he or she is watching is captured in the image.

[0025] On the other hand, the difficulty of making such a judgment may vary depending on the number of people captured in the video. That is, when the scene is quiet and the number of people captured in the video is very small (for example, one person), it is easy for the operator to grasp the behavior of the person and judge whether or not they are suspicious while the image capture device 100 is performing the patrol operation. On the other hand, when the number of people captured in the video increases, there are multiple targets for the operator to monitor, so it may be difficult for the operator to grasp the behavior of each person and make a judgment in the video in which the imaging range changes sequentially. That is, in order for the operator to properly complete the judgment in such a situation, it is necessary to stop the patrol operation of the image capture device 100 each time, which forces a complicated operation.

[0026] For this reason, the imaging device 100 of this embodiment controls so as to change the movement speed of the patrol operation based on the number of people detected in the video (hereinafter simply referred to as the number of detections). Specifically, when a number of people that would require a long time for the operator to make a decision are detected in the video, the imaging device 100 is controlled so as to change the movement speed of the patrol operation.

[0027] In one embodiment, the control of the operation speed may be performed using a preset threshold (detection number threshold). More specifically, when the number of detections in the video captured by the imaging unit 1 exceeds the detection number threshold, the operation speed of the patrol operation is controlled to be switched (slowed down) to a speed slower than the operation speed (reference speed) predetermined for normal patrol operation. This allows a person included in the imaging range to be captured in the video for a longer period of time. Also, when the number of detections in the video falls below the detection number threshold, the operation speed of the patrol operation is controlled not to be slowed down, that is, to be the reference speed.

[0028] In addition, when the number of detections is equal to the detection number threshold, the number of detections may be determined based on the control rule to be adopted. In this embodiment, when the number of detections is equal to the detection number threshold, the operation speed is controlled to be switched to a speed slower than the reference speed.

[0029] The detection number threshold and the speed after deceleration may be preset as fixed values ​​in the imaging device, but since the time required to grasp the contents of the video may vary depending on the ability and experience of the operator, the detection number threshold and the speed after deceleration may be set by the operator via the user terminal 200, for example. Here, a user interface 301 for setting the detection number threshold and the speed after deceleration in the user terminal 200 will be described with reference to Fig. 3. The data of the user interface 301 may be provided from the imaging device 100 to the user terminal 200 via the communication unit 5, for example, when a communication connection between the imaging device 100 and the user terminal 200 is established.

[0030] When the check box 302 of the user interface 301 is checked, an input field 303 for specifying a condition for decelerating the movement speed of the patrol movement (hereinafter referred to as a deceleration condition) is displayed on the user interface 301. As shown in the figure, the input field 303 includes a text box 311 for inputting a detection number threshold for each deceleration condition and a slider 312 for inputting a speed to be switched when the condition is met, in order to specify the deceleration condition of the patrol movement. In the example of FIG. 3, an example is shown in which two conditions are specified as the deceleration condition. In condition 1, it is specified that the movement speed is 30 (reference speed is 50) when two or more people are detected, and in condition 2, it is specified that the movement speed is 20 when three or more people are detected. Note that, although FIG. 3 illustrates an example of the input field 303 in which two types of deceleration conditions are specified, any number of deceleration conditions may be specified.

[0031] When a setting operation for the imaging device 100 regarding the deceleration conditions defined by the user interface 301 is performed on the user terminal 200, information on the deceleration conditions is transmitted from the user terminal 200 to the imaging device 100. When the communication unit 5 receives the information on the deceleration conditions, it stores the information in the memory 4, for example. The information is referred to when the control unit 8 executes an operation control process.

[0032] <<Motion Control Processing>> Hereinafter, the operation control process executed during patrol operation in the image capture device 100 of this embodiment will be described in detail with reference to the flowchart in Fig. 4. The process corresponding to the flowchart can be realized by the control unit 8 reading out a corresponding processing program stored in, for example, the non-volatile memory of the memory 4, and developing and executing the program in a volatile memory. This operation control process will be described as being started, for example, when the image capture device 100 is set to a state in which patrol operation is performed.

[0033] In S401, under the control of the control unit 8, the detection unit 6 detects a person (image) in the video captured by the imaging unit 1 for the current frame. When the detection unit 6 detects a person in the video related to the current frame, it outputs the number of detections to the control unit 8 as the detection result.

[0034] In S402, the control unit 8 judges whether or not the detection number of the detection result satisfies the deceleration condition of the operation speed. Information on the deceleration condition of the operation speed is stored in the memory 4, and the control unit 8 makes the judgment of this step by referring to the information. If the control unit 8 judges that the detection number satisfies the deceleration condition, it shifts the process to S403, and if it judges that it does not, it shifts the process to S404.

[0035] In S403, the control unit 8 transmits information on the operation speed specified in the satisfied deceleration condition to the operation control unit 7, and switches the operation speed of the drive unit 2. Here, when the detection number satisfies a plurality of deceleration conditions, the control unit 8 transmits information on the operation speed specified in the deceleration condition with the largest detection number threshold or the deceleration condition with the slowest specified speed to the operation control unit 7.

[0036] On the other hand, if it is determined in S402 that the detection number does not satisfy the deceleration condition, the control unit 8 determines in S404 whether or not the operation of the drive unit 2 was controlled at the operation speed specified in the deceleration condition in the most recent frame. If the control unit 8 determines that the operation of the drive unit 2 was controlled at the operation speed specified in the deceleration condition in the most recent frame, the control unit 8 proceeds to S405, and if it determines that the operation was not controlled at the operation speed, the control unit 8 proceeds to S406.

[0037] In S405, the control unit 8 transmits a control signal to the operation control unit 7 to cause the operation control unit 7 to perform a cyclic operation at a reference speed, and switches the operation speed of the drive unit 2.

[0038] Here, the operation control performed by the operation control unit 7 in the mode in which the two types of deceleration conditions shown in FIG. 3 are defined will be described with reference to FIG. 5. In the example shown in FIG. 5, the imaging unit 1 pans from FIG. 5(a) to FIG. 5(b) to FIG. 5(c) to FIG. 5(d). In the state of FIG. 5(a), no person is detected in the image of the current frame, and none of the deceleration conditions are satisfied, so the operation control unit 7 controls the drive unit 2 to perform the patrol operation at the reference speed. In the state of FIG. 5(b), two people are detected in the image of the current frame, and the deceleration condition of condition 1 is satisfied, so the operation control unit 7 controls the drive unit 2 to perform the patrol operation by switching to the operation speed of "30" defined in condition 1. In the state of FIG. 5(c), three people are detected in the image of the current frame, and the deceleration conditions of conditions 1 and 2 are satisfied, so the operation control unit 7 controls the drive unit 2 to perform the patrol operation by switching to the operation speed of "20" defined in condition 2. In the state of Fig. 5(d), only one person is detected in the image of the current frame, and again, none of the deceleration conditions are satisfied, so the operation control unit 7 controls the drive unit 2 to switch back to the reference speed and perform patrol operation. That is, the operation control unit 7 controls to decelerate the patrol operation when there is a transition from Fig. 5(a) to Fig. 5(b) and when there is a transition from Fig. 5(b) to Fig. 5(c), and to accelerate the patrol operation when there is a transition from Fig. 5(c) to Fig. 5(d).

[0039] Note that, although the example of FIG. 5 shows a patrol operation in which the imaging unit 1 pans, it is easily understood that the implementation of the present invention is not limited to this. As described above, the patrol operation may be configured to include pan, tilt, and zoom operations. For example, in a mode in which the patrol operation includes an operation of increasing the zoom magnification, the imaging range becomes narrower due to the zoom operation, so that the operation speed may be controlled to be slower than the operation speed specified for the deceleration condition. For example, when driving the zoom from the widest angle position to the most telephoto position, the operation control unit 7 may control the operation speed to be gradually decreased according to the zoom magnification so that the operation speed becomes half the operation speed specified for the corresponding deceleration condition when the zoom is moved to the most telephoto position.

[0040] In S406, the operation control unit 7 controls the drive unit 2 to perform a patrol operation at the currently set operation speed under the control of the control unit 8. More specifically, the operation control unit 7 determines which imaging direction to shift to in the current frame from the most recent imaging direction based on information that defines the transition of the imaging direction for a predetermined patrol operation, according to the currently set operation speed. The operation control unit 7 drives the drive unit 2 so that the imaging unit 1 captures an image in the determined imaging direction.

[0041] In S407, the control unit 8 judges whether the state of the image capturing device 100 is no longer in a state for performing a patrol operation. If the control unit 8 judges that the state of the image capturing device 100 is no longer in a state for performing a patrol operation, the control unit 8 completes this operation control process, and if the control unit 8 judges that the state for performing a patrol operation remains, the process returns to S401 and executes the process for the next frame.

[0042] As described above, according to the imaging device of this embodiment, the operation of the drive unit 2 can be controlled so as to output images whose contents are easy to understand during the patrol operation. That is, by controlling the operation speed of the patrol operation to be slower when the number of people detected in the image exceeds the detection number threshold, the period during which images capturing similar people are presented can be extended, and as a result, more time can be given to the operator to understand the contents of the images.

[0043] In this embodiment, the patrol operation is performed at a reference speed when the number of detections falls below the detection number threshold, and the control is performed to decelerate when the number of detections exceeds the threshold, but the implementation of the present invention is not limited to this. The control standard for the operation speed may be based on the situation where the number of detections is equal to or greater than the detection number threshold, and the control is performed to accelerate when the number of detections falls below the threshold. In other words, the operation speed in a situation where the number of detections is equal to or greater than the detection number threshold may be set as the reference speed, and the speed may be controlled to be changed to a speed different from the reference speed, or faster than the reference speed, in a situation where the number of detections is less than the threshold.

[0044] In addition, in the present embodiment, the imaging system has been described as being used for monitoring purposes, but the use of the present invention is not limited to this. The imaging system can also be used for presenting information to a switcher (an operator who selects a video to be distributed) in a mode in which a plurality of imaging devices 100 switch between and transmit the video being captured in video distribution or broadcasting. That is, in a mode in which a plurality of videos related to imaging devices 100 that are candidates for switching are presented to a switcher in order to distribute a video that appropriately conveys the atmosphere of the imaging environment, the present invention can be used to facilitate the switcher's understanding of the contents of each video. The imaging system can also be used for presenting a through image to an operator in a mode in which the imaging device 100 is remotely operated to capture a still image. That is, in a mode in which the video being captured by the imaging device 100 is presented to the operator as a through image, the present invention can be used to facilitate the operator's understanding of the situation of a subject included in the imaging range. The implementation of the present invention is not limited to these, and can be applied to any mode in which the imaging device 100 performs a predetermined series of operations including at least one of pan, tilt, and zoom.

[0045] [Embodiment 2] In the above embodiment, the operation speed is controlled according to only the number of detected people, but the present invention is not limited to this. For example, in the above embodiment, in the case where the present invention is used for evaluating the video image itself rather than for monitoring purposes, the operation speed may be controlled on the condition that the scene is one that is suitable for evaluation.

[0046] For example, in a case where the imaging device 100 is installed in a venue where a concert or a play is held, the light and dark are separated between the stage side and the audience seating side, and the atmosphere of the scene differs depending on which side is included in the imaging range. In this case, if the scene that the operator wants to focus on is the stage side, the operator is less interested in the video captured while the imaging device 100 is facing the audience seating side. In contrast, the operator is more interested in the video captured while the imaging device 100 is facing the stage side. Therefore, it is preferable to present for a long period of time, among the videos captured while the patrol operation is being performed, the video of the scene that tends to be of high interest to the operator, so in this embodiment, the operation speed is controlled on the condition that it is that scene.

[0047] In this embodiment, the scene is evaluated based on the luminance of the image captured by the imaging device. The luminance of the image can be determined based on the most frequent value when a histogram of luminance values ​​is generated for the image of the corresponding frame. The scene is evaluated based on the luminance by comparing the luminance of the scene with a predetermined threshold value for luminance evaluation (luminance threshold value).

[0048] The brightness threshold may be a fixed value that is set in advance in the imaging device, but since what kind of brightness scene is to be evaluated as suitable may differ depending on the use of the video, for example, the brightness threshold may be configured to be set by an operator via the user terminal 200. Here, a user interface 601 for setting deceleration conditions including brightness conditions in the user terminal 200 will be described with reference to FIG. 6. The data of the user interface 601 may be provided from the imaging device 100 to the user terminal 200 via the communication unit 5 when, for example, a communication connection between the imaging device 100 and the user terminal 200 is established. The user interface 601 includes an interface for specifying the brightness of a scene for changing the operation brightness of the patrol operation in addition to various interfaces constituting the user interface 301 shown in FIG. 3. That is, the user interface 601 makes it possible to additionally specify in which scene the deceleration is to be performed for the deceleration condition that determines the number of detections for decelerating the operation speed of the patrol operation.

[0049] When the check box 602 of the user interface 601 is checked, an input field 603 for specifying a scene brightness condition (hereinafter referred to as a brightness condition) for changing the operation speed of the patrol operation is displayed on the user interface 601. As shown in the figure, the input field 603 includes a slider 611 for inputting a brightness threshold value to specify the brightness of the scene for which the deceleration condition specified in the input field 303 is actually executed. The input field 603 also includes a drop-down list 612 for inputting which brightness range corresponds to the brightness condition, with the brightness threshold value specified by the slider 611 as a boundary. The drop-down list 612 may include, for example, "exceeding (above)", "equal to or greater than", "equal to or less than", and "less than" as options for the brightness threshold value. In the example of FIG. 6, an example is shown in which one deceleration condition is specified as the deceleration condition. In condition 1, it is specified that the operation speed is 30 when two or more people are detected, and further, it is specified that the deceleration control is performed on the condition that the brightness of the scene is 128 or more for the deceleration condition. Although FIG. 5 illustrates an example of the user interface 601 in which one type of deceleration condition and one type of brightness condition are defined, any number of combinations of deceleration conditions and brightness conditions may be defined.

[0050] When a setting operation for the imaging device 100 regarding the deceleration conditions and brightness conditions defined by the user interface 601 is performed on the user terminal 200, information on the conditions is transmitted from the user terminal 200 to the imaging device 100. When the communication unit 5 receives the information on the deceleration conditions and brightness conditions, it stores the information in the memory 4, for example. The information is referred to when the control unit 8 executes an operation control process.

[0051] <<Motion Control Processing>> Hereinafter, the operation control process executed during the patrol operation in the imaging device 100 of this embodiment will be described in detail with reference to the flowchart in Fig. 7. The process corresponding to this flowchart can be realized by the control unit 8 reading out the corresponding processing program stored in, for example, the non-volatile memory of the memory 4, and developing and executing it in the volatile memory. This operation control process will be described as being started, for example, when the imaging device 100 is set to a state in which the patrol operation is performed. In the description of the operation control process of this embodiment, steps that perform the same processes as those in the operation control process of embodiment 1 are given the same reference numbers and their description will be omitted, and only the steps that perform the processes unique to this embodiment will be described below.

[0052] In S701, the image processing unit 3, under the control of the control unit 8, derives the luminance of a scene based on the video captured by the imaging unit 1 for the current frame. After deriving the luminance of the scene related to the current frame, the image processing unit 3 outputs the derived result to the control unit 8.

[0053] In S702, the control unit 8 judges whether or not the luminance of the scene related to the current frame satisfies the luminance condition. Information on the luminance condition is stored in the memory 4, and the control unit 8 makes the judgment of this step by referring to the information. If the control unit 8 judges that the luminance of the scene related to the current frame satisfies the luminance condition, it shifts the process to S401, and if it judges that the luminance does not satisfy the luminance condition, it shifts the process to S406.

[0054] For example, consider a case where the imaging device 100 is installed in an imaging environment including a bright area and a dark area as shown in FIG. 8. In the example of FIG. 8(a), the imaging device 100 mainly captures a dark area with low brightness in the imaging range, and in the example of FIG. 8(b), the imaging device 100 mainly captures a bright area with high brightness in the imaging range. Here, in the aspect in which the deceleration condition and the brightness condition shown in FIG. 6 are specified, the condition for decelerating the operation speed of the patrol operation is that the brightness is 128 or more and the number of detections is 2 or more, so FIG. 8(b) is the subject of the deceleration control, and FIG. 8(a) is not the subject of the deceleration control. On the other hand, when "less than" is set in the drop-down list 612 in the input field 603 of FIG. 6, the condition for decelerating the operation speed of the patrol operation is that the brightness is less than 128 and the number of detections is 2 or more. Therefore, FIG. 8(a) is the subject of the deceleration control, and FIG. 8(b) is not the subject of the deceleration control.

[0055] As described above, according to the imaging device of the present embodiment, the operation of the driving unit 2 can be controlled so as to output an image whose contents are easy to understand during the patrol operation. That is, when the brightness of the scene in the imaging range satisfies a predetermined brightness condition and the number of people detected in the image exceeds the detection number threshold, the operation speed of the patrol operation is controlled to be slow, so that the period during which the image of the desired scene is presented can be extended. More specifically, only when the scene has the brightness desired by the operator and the number of people captured in the image is large, an image that allows the operator time to understand the contents can be provided, thereby realizing efficient image evaluation.

[0056] In this embodiment, the operation speed is controlled according to the number of people detected, provided that the brightness of the scene satisfies a predetermined brightness condition, but the present invention is not limited to this. What kind of scene is evaluated as being favorable depends on the video usage policy, so for example, the color distribution of the scene or the degree of excitement based on the sound included in the video may be evaluated.

[0057] [Variation 1] In the above-mentioned second embodiment, the operation speed of the patrol operation is changed under the condition that the brightness of the scene captured in the video meets the predetermined brightness condition and the number of detections in the video exceeds the detection number threshold, but the implementation of the present invention is not limited to this. For example, when the brightness of the scene meets the brightness condition during the execution of the patrol operation, the operation speed may be controlled regardless of the number of people detected.

[0058] [Embodiment 3] In the above embodiment, the aspect of controlling the patrol operation by switching to a predetermined operation speed according to the number of detected persons has been described, but the embodiment of the present invention is not limited to this. Even if the operation speed of the patrol operation is switched to a low speed, the operation speed after switching is not necessarily suitable for the movement of the person captured in the imaging range. For this reason, depending on the moving speed of the person, a situation may occur in which the image of the person that the operator is gazing at is not captured in the subsequent frame. Alternatively, a situation may occur in which the image of the person that the operator is gazing at is captured in the video for a short period of time, and the operator is unable to fully grasp the content. This is particularly evident when a person moving in the opposite direction to the rotation direction of the imaging device 100 is present within the imaging range. In this embodiment, an aspect will be described in which, when the operation speed of the patrol operation is switched to a low speed, the relative moving speed of each person captured in the video relative to the imaging device 100 is detected, and the operation speed of the patrol operation is further controlled according to the moving speed.

[0059] In the imaging device 100 of this embodiment, the relative moving speed of a person captured in a video is derived based on the magnitude of the motion vector of the image of the person in the captured video. More specifically, the position of the image of each person is specified for each frame of the video, and the image of the same person is matched between frames to derive the motion vector of each person's image. The derivation of the motion vector of each person can be performed, for example, by using a known method such as a block matching method for an area recognized as a person in an image related to a frame. The execution of the control of the motion speed based on the motion vector may be determined based on whether or not the person is captured within the imaging range of the video obtained after a predetermined frame by performing a patrol operation at the switching motion speed. In other words, the execution of the speed control based on the motion vector may be determined based on whether or not the position of the person after a predetermined frame, which is derived based on the motion vector, is included within the angle of view of the video obtained in that frame.

[0060] The execution or non-execution of the control of the motion speed based on the motion vector may be configured to be set by an operator via a user interface including a check box 901 in addition to the user interface 301 as shown in FIG. 9. The check box 901 is arranged and presented in the input field 303 when the check box 302 is checked. When the check box 901 is checked, a deceleration condition for deriving the motion vector of each person captured in the video and controlling the motion speed based on the motion vector when the corresponding deceleration condition is satisfied can be set in the imaging device 100. That is, when the check box 901 is checked, when the detection number exceeds the detection number threshold, the patrol operation is basically controlled to the motion speed specified by the slider 312, but a deceleration condition for adaptively controlling the motion speed according to the motion of the person in the video is set. Note that FIG. 9 illustrates an example of a user interface in which one type of deceleration condition is specified, but it goes without saying that any number of deceleration conditions may be specified.

[0061] <<Motion Control Processing>> Hereinafter, the operation control process executed during the patrol operation in the imaging device 100 of this embodiment will be described in detail with reference to the flowchart in Fig. 10. The process corresponding to the flowchart can be realized by the control unit 8 reading out the corresponding processing program stored in, for example, the non-volatile memory of the memory 4, and executing it by expanding it in the volatile memory. This operation control process will be described as being started, for example, when the imaging device 100 is set to a state in which the patrol operation is performed. In the description of the operation control process of this embodiment, steps that perform the same processes as those in the operation control process of embodiment 1 are given the same reference numbers and their description will be omitted, and only the steps that perform the processes unique to this embodiment will be described below.

[0062] If it is determined in S402 that the number of detections satisfies the deceleration condition, the control unit 8 derives a motion vector for each person detected in the video related to the current frame in S1001. The motion vector (relative moving speed) for each person is derived based on the difference in two-dimensional coordinates of an image estimated to be the same person between the video related to the most recent frame and the video related to the current frame, for example.

[0063] In S1002, the control unit 8 judges whether or not a person in the image related to the current frame is captured within the imaging range after operating for a predetermined time (for example, the number of frames corresponding to one second) at the operating speed specified in the satisfied deceleration condition. That is, the control unit 8 judges whether or not the person detected in S401 can be captured within the angle of view for a predetermined time when the person is caused to move around at the operating speed specified in the deceleration condition. The judgment is made based on the detected position of the person in the image related to the current frame and the motion vector related to each person. Here, since the detected number of people are captured in the image, the average or maximum of those motion vectors can be used for the judgment. If the control unit 8 judges that the person in the image related to the current frame is captured within the imaging range after operating for a predetermined time, the control unit 8 shifts the process to S403, and if the control unit 8 judges that the person is not captured, the control unit 8 shifts the process to S1003.

[0064] In S1003, the control unit 8 derives a motion speed for keeping the person in the image related to the current frame captured within the imaging range for a predetermined time. The motion speed derived in this step is determined according to the direction of the motion vector and the turning direction of the patrol motion. For example, if the person is moving in the opposite direction to the previous direction, the control unit 8 derives a motion speed that is lower than the motion speed specified in the deceleration condition. That is, the control unit 8 specifies how much the detected person should be decelerated in order to keep the person captured within the angle of view for a predetermined time, and derives the motion speed.

[0065] In S1004, the control unit 8 transmits the information on the operation speed derived in S1003 to the operation control unit 7, and switches the operation speed of the drive unit 2. In an embodiment in which a minimum speed and a maximum speed are set for the operation speed of the patrol operation, the control unit 8 may change the operation speed information to be transmitted so that the operation speed falls within these speed ranges. Specifically, when the control unit 8 derives an operation speed below the minimum speed in S1003, it transmits the minimum speed information to the operation control unit 7.

[0066] As described above, according to the imaging device of this embodiment, the patrol operation is controlled according to the behavior of the person captured in the image, so that imagery that is easier to understand can be output during the patrol operation.

[0067] [Variation 2] In the above-mentioned embodiment and modified example, a person included in the video is detected and the movement speed of the patrol movement is controlled based on the detection result. However, the present invention is not limited to this embodiment. It goes without saying that the movement speed may be controlled based on the detection result of detecting any object (e.g., an animal, a ball, a car, etc.) included in the video. In this case, information such as a feature amount for detecting each object is stored in the memory 4 of the imaging device 100 and used for the operation of the detection unit 6. In addition, the user interface for specifying the deceleration condition provided to the user terminal 200 may be configured so that the type of object can be selected from, for example, a drop-down list.

[0068] [Variation 3] In the above-mentioned embodiment and modified example, when the control condition of the motion speed is satisfied, the motion speed is switched to a speed slower than the reference speed, but the motion speed may be changed in a stepwise manner. For example, in order to prevent a sudden change in the motion speed, the motion speed may be controlled to approach the motion speed after the switching in sequence so that the difference between the motion speed in the most recent frame and the motion speed set for the current frame does not exceed a predetermined value. Alternatively, in an aspect in which the deceleration condition is not set, the motion speed may be controlled to change in a stepwise manner simply according to the number of detections in the video during the execution of the patrol operation. That is, the motion control unit 7 may control the motion speed of the patrol operation to decelerate as the number of detections increases, and to approach the reference speed as the number of detections decreases.

[0069] [Variation 4] In the above-mentioned embodiment and modified example, the patrol operation is set in a manner that can be repeatedly executed, but the implementation of the present invention is not limited to this. The patrol operation may be a series of operations that are defined to change at least one of pan, tilt, and zoom, and does not need to be set so that the pan angle, tilt angle, and zoom magnification are the same at the start and end.

[0070] [Variation 5] In the above-described embodiment and modified example, the imaging device 100 is provided with a functional configuration for controlling the operation speed of the patrol operation, but the present invention is not limited to this. The function may be provided as a control device outside the imaging device 100, and the imaging device 100 may control the patrol operation based on a control signal from the control device.

[0071] [Summary of the embodiment and modifications] The disclosure of this specification includes the following control device, imaging device, control method, and program. (Item 1) A control device for controlling an imaging device having a driving means for performing at least one of pan, tilt, and zoom operations, comprising: an acquisition means for acquiring an image captured by an imaging means of the imaging device; a detection means for detecting an object included in the video captured by the capture means; A control means for controlling the operation of the driving means; having The control device according to claim 1, wherein the control means changes an operating speed of the drive means based on a detection result by the detection means during the process of causing the drive means to perform a predetermined series of operations. (Item 2) 2. The control device according to item 1, wherein the control means changes the operating speed of the drive means based on the number of detected objects. (Item 3) 3. The control device according to item 2, wherein the control means makes the operating speed of the drive means slower than a predetermined operating speed when the number of detections of the object exceeds a detection number threshold. (Item 4) 4. The control device according to item 3, wherein the control means causes the operating speed of the drive means to be the predetermined operating speed when the number of detected objects falls below the detection number threshold. (Item 5) 5. The control device according to any one of items 2 to 4, wherein the control means changes the operating speed of the drive means in a stepwise manner in response to the number of detected objects. (Item 6) the detection means detects a relative moving speed of the object in the video; 6. The control device according to any one of items 1 to 5, wherein the control means changes an operating speed of the drive means based on the relative moving speed of the object. (Item 7) 7. The control device according to item 6, wherein the control means changes the operating speed of the drive means based on the relative moving speed of the object so as to capture the object within the angle of view. (Item 8) Further comprising an evaluation means for evaluating the scene captured in the video, The control means changes the operating speed of the driving means based on the evaluation of the scene by the evaluation means and the detection result by the detection means in the process of causing the driving means to perform the series of operations. 8. The control device according to any one of items 1 to 7, (Item 9) said evaluating means evaluating the luminance of said scene; The control device described in item 8, characterized in that the control means changes the operating speed of the drive means based on the detection result by the detection means, on the condition that the brightness of the scene falls under a predetermined brightness condition. (Item 10) 10. The control device according to item 9, wherein the control means does not change the operating speed of the drive means when the luminance of the scene does not meet the luminance condition. (Item 11) 11. The control device according to any one of items 1 to 10, wherein the predetermined series of operations are configured to be repeatedly executable. (Item 12) An imaging means; A driving means for performing at least one of pan, tilt and zoom operations; A control device according to any one of items 1 to 11, An imaging device comprising: (Item 13) A method for controlling an imaging device having a driving unit that performs at least one of pan, tilt, and zoom operations, comprising the steps of: an acquisition step of acquiring an image captured by an imaging means of the imaging device; a detection step of detecting an object included in the video captured in the capture step; a control step of controlling the operation of the driving means; having A control method, characterized in that in the control step, the operating speed of the drive means is changed based on the detection result in the detection step during the process of causing the drive means to perform a predetermined series of operations. (Item 14) A program for causing a computer to function as each of the means of the control device according to any one of items 1 to 11.

[0072] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0073] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0074] 1: imaging unit, 2: driving unit, 3: image processing unit, 6: detection unit, 7: operation control unit, 8: control unit, 100: imaging device

Claims

1. A control device for controlling an imaging device having a driving means for performing at least one of pan, tilt, and zoom operations, comprising: an acquisition means for acquiring an image captured by an imaging means of the imaging device; a detection means for detecting an object included in the video captured by the capture means; A control means for controlling the operation of the driving means; having The control device according to claim 1, wherein the control means changes an operating speed of the drive means based on a detection result by the detection means during the process of causing the drive means to perform a predetermined series of operations.

2. 2. The control device according to claim 1, wherein the control means changes the operating speed of the drive means based on the number of detected objects.

3. 3. The control device according to claim 2, wherein the control means makes the operating speed of the drive means slower than a predetermined operating speed when the number of detected objects exceeds a detection number threshold.

4. 4. The control device according to claim 3, wherein the control means causes the movement speed of the drive means to be the predetermined movement speed when the number of detected objects falls below the detection number threshold.

5. 3. The control device according to claim 2, wherein the control means changes the operating speed of the driving means in a stepwise manner in response to the number of detected objects.

6. the detection means detects a relative moving speed of the object in the video; The control device according to claim 1 , wherein the control means changes an operating speed of the drive means based on the relative moving speed of the object.

7. 7. The control device according to claim 6, wherein the control means changes an operating speed of the drive means based on the relative moving speed of the object so that the object is captured within the angle of view.

8. Further comprising an evaluation means for evaluating the scene captured in the video, The control means changes the operating speed of the driving means based on the evaluation of the scene by the evaluation means and the detection result by the detection means in the process of causing the driving means to perform the series of operations. The control device according to claim 1 .

9. said evaluating means evaluating the luminance of said scene; 9. The control device according to claim 8, wherein the control means changes the operating speed of the drive means based on the detection result by the detection means, on condition that the luminance of the scene meets a predetermined luminance condition.

10. 10. The control device according to claim 9, wherein the control means does not change the operating speed of the drive means when the luminance of the scene does not meet the luminance condition.

11. 2. The control device according to claim 1, wherein the predetermined series of operations is configured to be repeatedly executable.

12. An imaging means; A driving means for performing at least one of pan, tilt and zoom operations; A control device according to any one of claims 1 to 11; An imaging device comprising:

13. A method for controlling an imaging device having a driving unit that performs at least one of pan, tilt, and zoom operations, comprising the steps of: an acquisition step of acquiring an image captured by an imaging means of the imaging device; a detection step of detecting an object included in the video captured in the capture step; a control step of controlling the operation of the driving means; having A control method, characterized in that in the control step, the operating speed of the drive means is changed based on the detection result in the detection step during the process of causing the drive means to perform a predetermined series of operations.

14. A program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 11.

Citation Information

Patent Citations

  • Display device, display method and program

    JP2017125974A