OBJECT TRACKING DEVICE, OBJECT TRACKING METHOD, AND PROGRAM

Abstract

Reducing the resources required when tracking objects using high-resolution images. [Solution] The object tracking device (10) includes an object identification information acquisition unit (110), an image processing unit (120), and a control unit (130). The object identification information acquisition unit (110) acquires object identification information that identifies the object to be tracked. The image processing unit (120) performs image processing on a processing target area, which is a partial area of ​​an image acquired from an imaging device (20), to detect the object to be tracked and track the detected object. The control unit (130) controls the imaging range of the imaging device (20) so that the object to be tracked is included in the processing target area.

Description

[Technical field]

[0001] The present invention relates to a technique for detecting and tracking a specific object using an image. [Background technology]

[0002] Examples of techniques for tracking a specific object in an image are disclosed in, for example, Patent Documents 1 to 3 below. Patent Document 1 below discloses a technique for manually (user operation) or automatically tracking a subject when cutting out a partial area including the subject to be tracked and processing it as a cut-out video. Patent Document 2 below discloses a technique for processing the entire area of ​​an image of an imaging device attached to a shovel to identify a partial area where a person is likely to be present, and further processing the partial area to determine whether or not a person is present in the area. Patent Document 3 below discloses a technique for detecting a moving object from an image generated by an imaging unit whose imaging direction can be changed by panning and tilting, and performing panning and tilting control so that the moving object is located near the center of the image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2016 / 002228 [Patent Document 2] JP 2017-151815 A [Patent Document 3] JP 2016-092606 A Summary of the Invention [Problem to be solved by the invention]

[0004] High-performance imaging devices can generate images with extremely high resolutions, such as 4K or 8K. High-resolution images have a huge data volume. Therefore, when processing high-resolution images, the resources required for the image processing machine are large.

[0005] The present invention has been made in view of the above-mentioned problems. It is an object of the present invention to provide a technique for reducing resources required when performing object tracking using high-resolution images. [Means for solving the problem]

[0006] The object tracking device of the present invention comprises: An object identification information acquisition means for acquiring object identification information for identifying an object to be tracked; an image processing means for detecting an object to be tracked by performing image processing on a processing target area, which is a partial area of ​​an image acquired from an imaging device, and tracking the detected object; a control means for controlling an imaging range of the imaging device so that the object to be tracked is included in the processing target area; Equipped with.

[0007] The object tracking method of the present invention includes: The computer Acquire target identification information that identifies the object to be tracked; performing image processing on a processing target area, which is a partial area of ​​an image acquired from an imaging device, to detect an object to be tracked and track the detected object; controlling an imaging range of the imaging device so that the object to be tracked is included in the processing target area; This includes:

[0008] The program of the present invention causes a computer to execute the object tracking method described above. Effect of the Invention

[0009] According to the present invention, it is possible to reduce the resources required when performing object tracking using high-resolution images. [Brief description of the drawings]

[0010] The above objects, as well as other objects, features and advantages, will become more apparent from the following preferred embodiments and the accompanying drawings.

[0011] [Figure 1] FIG. 2 is a diagram for explaining an overview of a process executed by an object tracking device according to the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an example of a functional configuration of an object tracking device according to the first embodiment. [Diagram 3] FIG. 10 is a diagram illustrating an example of the relationship between a processing target area and a predetermined area. [Figure 4] FIG. 2 is a block diagram illustrating a hardware configuration of the object tracking device. [Diagram 5] 4 is a flowchart illustrating a flow of processing executed by the object tracking device of the first embodiment. [Figure 6] 4 is a flowchart illustrating a flow of processing executed by the object tracking device of the first embodiment. [Figure 7] 4 is a flowchart illustrating a flow of processing executed by the object tracking device of the first embodiment. [Figure 8] 11A and 11B are diagrams illustrating an example of an output from an image processing unit based on a result of an object detection process. [Figure 9] 11 is a diagram illustrating an example of a state in which a position designation input is performed on an image. FIG. [Figure 10] 11A and 11B are diagrams illustrating an example of an output from an image processing unit based on a result of an object detection process. [Figure 11] 11 is a flowchart illustrating a process performed by the object tracking device when the object to be tracked is lost. [Figure 12] 10 is a diagram illustrating a flow of acquiring target object position information when a tracking target object is lost; [Figure 13] 10 is a diagram illustrating a flow of acquiring target object position information when a tracking target object is lost; [Figure 14] FIG. 11 is a diagram illustrating an example of a functional configuration of an object tracking device according to a second embodiment. [Figure 15]10 is a flowchart illustrating a flow of a recording process executed by an object tracking device of a second embodiment. [Figure 16] FIG. 13 is a block diagram showing an example of a functional configuration of an object tracking device according to a third embodiment. [Figure 17] 13 is a flowchart illustrating the flow of a report generation process executed by the object tracking device of the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all drawings, similar components are given similar reference numerals and their description will be omitted as appropriate. In addition, unless otherwise specified, in each block diagram, each block represents a functional configuration, not a hardware configuration.

[0013] [Summary of the Invention] An overview of the present invention will be described with reference to Fig. 1. Fig. 1 is a diagram for explaining an overview of processing executed by an object tracking device 10 according to the present invention. In the example of Fig. 1, the object tracking device 10 is communicably connected to an imaging device 20 and a user terminal 30 via a network (not shown). The imaging device 20 is, for example, a surveillance camera installed in a city or inside a building. The user terminal 30 is a terminal used by a person (hereinafter, also referred to as a "user") who checks images generated by the imaging device 20.

[0014] In response to acquisition of information that identifies an object to be tracked (hereinafter also referred to as "object identification information"), the object tracking device 10 performs image processing to track an object identified by the object identification information on images acquired from the imaging device 20. The object tracking device 10 tracks a specific object among multiple images (two or more consecutive frame images) acquired from the imaging device 20 using a known object tracking algorithm.

[0015] Here, the object tracking device 10, as shown in FIG. 1, detects an entire area A of an image acquired from an imaging device 20. EInstead, the processing target area A P In other words, the object tracking device 10 processes a processing target area A, which is a part of the image acquired from the imaging device 20. P The result of this image processing is output on the display of the user terminal 30 together with the image captured by the imaging device 20.

[0016] Note that the object to be tracked is in the processing target area A P If the object moves outside the processing target area A, the object tracking device 10 will not be able to track the object. P The imaging range of the imaging device 20 is controlled using the position of the object to be tracked (position in the image coordinate system) so that the imaging range of the imaging device 20 is included in the position of the object to be tracked. The object tracking device 10 can change the imaging range of the imaging device 20 by controlling the zoom ratio of the imaging device 20, the orientation of the imaging device 20, or the operation of a mechanism (not shown) that controls the position of the imaging device 20 using a control signal.

[0017] <Actions and Effects> As described above, in the present invention, when information (object identification information) that identifies an object to be tracked is acquired, image processing is executed to track the object identified by the object identification information. This image processing is executed on a processing target area A, which is a part of the image acquired from the imaging device 20. P It is performed on the image. By limiting the area to be processed to a part of the image, it is possible to reduce the computational costs involved in image processing. Furthermore, by reducing the computational costs, it becomes possible to process high-resolution images with little (or no) delay, even if the performance of the computer used for image processing is low.

[0018] However, when the area to be subjected to image processing is restricted, a problem occurs in that the area in which the object can be tracked in the image becomes small. Therefore, in the present invention, the object to be tracked is restricted to the processing area A. PThe zoom ratio, orientation, or position of the imaging device 20 is controlled so that the range in which the object can be tracked is included in the range of the object that can be tracked. This effectively expands the range in which the object can be tracked. In other words, according to the present invention, it is possible to obtain the effect of reducing the computational cost required for image processing while suppressing the influence on the range in which the object can be tracked.

[0019] [First embodiment] <Example of functional configuration> 2 is a diagram showing an example of the functional configuration of the object tracking device 10 in the first embodiment. As shown in FIG. 2, the object tracking device 10 includes an object identification information acquisition unit 110, an image processing unit 120, and a control unit .

[0020] The object identification information acquisition unit 110 acquires object identification information. The "object identification information" is information for identifying a tracked object included in an image acquired from the imaging device 20 (information for uniquely identifying a tracked object included in an image). The object identification information is information that can be extracted from an image, such as a feature amount indicating a characteristic unique to the tracked object (e.g., a color or shape unique to the object). The object identification information may also be information indicating an image area from which a feature amount is to be extracted (e.g., information specifying a position or area on an image).

[0021] The image processing unit 120 processes the processing target area A P In other words, the image processing unit 120 executes image processing for detecting an object to be tracked that is specified by the object specifying information and tracking the detected object. P For areas other than the processing target area A, image processing for detecting and tracking the object to be tracked is not performed. Therefore, even if an object is included in the image acquired from the imaging device 20, the object is not included in the processing target area A. P If an object is located in a region outside of the target area, then the object will not be detected and tracked by the image processor 120.

[0022] << Processing area A P About >> Here, the processing target area A Pis a partial area of ​​an image, as shown in FIG. P Preferably, the processing target area A includes the central part of the image (the area near the center of the image). P The shape of the processing target area A is not particularly limited. P is not limited to a rectangular shape as exemplified in the drawings of this specification, but may have other shapes such as a polygon or a circle (ellipse).

[0023] As an example, the processing target area A P The shape of the processing target area A can be set arbitrarily. For example, the object tracking device 10 receives an input specifying an area to be processed or an area not to be processed via the user terminal 30, and determines a shape of the processing target area A based on the input. P In this case, the processing area A calculated based on the specified input may be defined. P The position coordinates of are stored (set) in a memory or the like.

[0024] As another example, the processing target area A P The size of the processing target area A may be determined in advance. P may be set to a size that ensures sufficient processing speed (for example, a size equivalent to a VGA (Video Graphics Array) (approximately 300,000 pixels)). In this case, the processing target area A P The position coordinates of the object are stored (set) as fixed values ​​in a memory or the like.

[0025] In the present invention, in order to reduce the computational cost of image processing, P Therefore, if the image processing device has sufficient resources, the processing target area A P Even if the area A becomes larger to some extent, the processing speed experienced by the user is not affected much. P The size of the processing target area A PThe size of the processing target area A may be determined based on the size of the resources that can be allocated to image processing for the processing target area A. Specifically, the object tracking device 10 first acquires information on its own surplus resources, and then, based on the information, P Then, the object tracking device 10 determines the size of the resources that can be allocated to image processing for the processing target area A based on the size of the resources that can be allocated. P The size (position coordinates) of the object is determined, and the position coordinates are stored (set) in a memory or the like.

[0026] The control unit 130 determines whether the object to be tracked, which is identified by the object identification information, is in the processing target area A. P The control unit 130 uses multiple images to obtain the movement (displacement of the detection position) of the object to be tracked, and operates a mechanism (not shown) for controlling the image capture range of the image capture device 20 in accordance with the movement, thereby determining whether the object to be tracked is included in the processing target area A. P The imaging range of the imaging device 20 can be controlled so as to be included in the range.

[0027] <<Specific example of mechanism for controlling the imaging range of the imaging device 20>> Specific examples of the "mechanism that controls the imaging range of the imaging device 20" include a mechanism that controls the zoom of the imaging device 20, an electric pan head that controls the orientation of the imaging device 20, an electric slider that controls the shooting position of the imaging device 20, etc.

[0028] <<Specific example of control by control unit 130>> The control unit 130 controls the imaging range of the imaging device 20 by operating a mechanism for controlling the imaging range of the imaging device 20 as described above. As an example, the control unit 130 can change the imaging range of the imaging device 20 by operating a mechanism for controlling the zoom of the imaging device 20 according to the movement of the object to be tracked on the image. As another example, when the imaging device 20 is mounted on an electric pan head (mechanism for controlling the orientation of the imaging device 20) not shown, the control unit 130 can change the imaging range of the imaging device 20 by operating the electric pan head according to the movement of the object to be tracked on the image. As yet another example, when the imaging device 20 is mounted on an electric slider (mechanism for controlling the position of the imaging device 20) not shown, the control unit 130 can change the imaging range of the imaging device 20 by operating the electric slider according to the movement of the object to be tracked on the image. The control unit 130 may also control the imaging range of the imaging device 20 by combining the operations on the multiple mechanisms exemplified above.

[0029] From the viewpoint of user visibility, the control unit 130 is preferably configured to control the imaging range of the imaging device 20 so that the object to be tracked is located at a point that is easily visible to the user (for example, near the center of the image). P The imaging range of the imaging device 20 is controlled so that the object to be tracked is included in a predetermined area (e.g., FIG. 3) that is a part of the processing target area A. P 3 is a diagram illustrating an example of the relationship between the predetermined area a and the processing target area A. P and the entire area A of the image E The information indicating the position coordinates of the predetermined area a is set to include the center O of the processing target area A P When the predetermined area a is set, the area is stored in the memory of the object tracking device 10. The control unit 130 determines whether the object to be tracked (in this figure, the vehicle V 13, the control unit 130 controls the operation of a mechanism that adjusts the zoom ratio, orientation, position, or the like of the imaging device 20 so that the imaging range of the imaging device 20 moves in the upper right direction.

[0030] [Hardware configuration example] The object tracking device 10 may be realized by hardware (e.g., hardwired electronic circuits, etc.) that realizes each functional component, or may be realized by a combination of hardware and software (e.g., a combination of electronic circuits and a program that controls the electronic circuits, etc.). The case where the object tracking device 10 is realized by a combination of hardware and software will be further described below.

[0031] FIG. 4 is a block diagram illustrating the hardware configuration of the object tracking device 10. As shown in FIG.

[0032] The object tracking device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

[0033] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0034] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0035] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0036] The storage device 1040 is an auxiliary storage device realized by a hard disk drive (HDD), a solid state drive (SSD), a memory card, a read only memory (ROM), or the like. The storage device 1040 stores program modules that realize each function of the object tracking device 10 (such as the target identification information acquisition unit 110, the image processing unit 120, and the control unit 130). The processor 1020 loads each of these program modules onto the memory 1030 and executes them, thereby realizing each function corresponding to each program module.

[0037] The input / output interface 1050 is an interface for connecting the object tracking device 10 and the peripheral devices 15. The peripheral devices 15 include, for example, input devices such as a keyboard and a mouse, and output devices such as a display (touch panel display) and a speaker. The object tracking device 10 is connected to these peripheral devices 15 via the input / output interface 1050.

[0038] The network interface 1060 is an interface for connecting the object tracking device 10 to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The network interface 1060 may be connected to the network wirelessly or by wire. The object tracking device 10 is communicably connected to external devices such as an imaging device 20 and a user terminal 30 via the network interface 1060. The imaging device 20 is, for example, a camera equipped with a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The user terminal 30 is a terminal used by a person (user) who checks images generated by the imaging device 20. The user terminal 30 is a stationary PC (Personal Computer) or a mobile terminal (such as a smartphone or a tablet terminal). The user terminal 30 is not particularly limited.

[0039] 4 is merely an example, and the hardware configuration of the object tracking device 10 is not limited to the example of FIG. 4. For example, a part or all of the functional processing units of the object tracking device 10 may be provided in the imaging device 20 (a so-called intelligent camera equipped with a CPU for performing image processing and peripheral device control) or the user terminal 30. In addition, when all of the functional processing units of the object tracking device 10 are provided in the imaging device 20 or the user terminal 30, the object tracking device 10 does not need to be present between the imaging device 20 and the user terminal 30.

[0040] [Processing flow] Hereinafter, the processing executed by the object tracking device 10 of the present embodiment will be described with reference to the drawings. Figures 5 to 7 are flowcharts illustrating the flow of the processing executed by the object tracking device 10 of the first embodiment.

[0041] First, the object tracking device 10 communicates with the imaging device 20 via the network interface 1060 and acquires an image generated by the imaging device 20 (S102).

[0042] After acquiring an image from the imaging device 20, the image processing unit 120 processes the processing target area A of the image generated by the imaging device 20. P (S104). The "object detection process" refers to image processing that detects an object (e.g., a person, a vehicle, a motorcycle, etc.) that is predefined as a detection target. The image processing unit 120 can detect a specific object from an image acquired from the imaging device 20 using an object detector stored in the memory 1030 or the storage device 1040. The object detector is constructed to be able to detect a specific object (e.g., a person, a vehicle, a motorcycle, etc.) by machine learning, for example. The object detector may also be constructed to be able to detect a specific object by a rule base. The object detector may also be constructed to be able to distinguish and detect an object having a specific attribute from among objects of the same type. Specific examples of the "attribute" include, but are not limited to, the following: -Person attributes: age, gender, clothing characteristics, possessions, etc. Vehicle / motorcycle attributes: vehicle color, model, shape and size, license plate number, number of passengers, etc. The object detector may also be constructed to be capable of determining attributes related to the behavior of an object (e.g., staggering, behaving suspiciously, etc.) and detecting a specific object having those attributes.

[0043] After executing the object detection process, the image processing unit 120 superimposes the object detection result obtained by the process on the image acquired from the imaging device 20 as shown in FIG. 8, for example, and outputs the result to the display of the user terminal 30 (S106). FIG. 8 is a diagram showing an example of the output of the image processing unit 120 based on the result of the object detection process. In the example of FIG. 8, the image processing unit 120 P The object detected in 1 and V 2 ) The object detection result (rectangular frame F 1 and F 2 ) are superimposed and output to the display of the user terminal 30. P Objects (vehicles V) located outside the 3 ), the image processing unit 120 cannot detect the object. 3 Regarding vehicle V 1 and V 2 The frame F that is added to 1 and F 2 Such information is not added.

[0044] Then, the object tracking device 10 determines whether or not the object identifying information has been acquired by the object identifying information acquisition unit 110 (S108). If the object identifying information has not yet been acquired (S108: NO), an object identifying information acquisition process is executed. The flow of the object identifying information acquisition process is illustrated in FIG. 6. On the other hand, if the object identifying information has already been acquired (S108: YES), an object tracking process is executed. The flow of the object tracking process is illustrated in FIG. 7.

[0045] <<Object specific information acquisition process>> The object identification information acquisition unit 110 judges whether or not an input (position designation input) designating the position of the object to be tracked has been received (S110). For example, when a user selects the position of the object to be tracked on an image displayed on the display of the user terminal 30, information indicating the selected position (position on the image coordinates) is input to the object identification information acquisition unit 110 (e.g., FIG. 9). FIG. 9 is a diagram illustrating a state in which a position designation input is performed on an image. The user can designate any position on the image using an input device (mouse, keyboard, touch panel, etc.) of the user terminal 30. For example, the user can operate a pointer P illustrated in this figure using a mouse or a touch panel to select any position on the image.

[0046] When the user performs position designation input, the image processing unit 120 P The image processing unit 120 may superimpose information indicating the processing target area A on the image acquired from the imaging device 20 and display the information on the display of the user terminal 30. For example, the image processing unit 120 may superimpose information indicating the processing target area A on the image acquired from the imaging device 20 on the display of the user terminal 30. P In this way, the processing target area A P By visualizing in the image, the user can easily determine whether or not an object is included in the area where image processing is performed.

[0047] Here, the processing target area A P In order to track an object located outside the processing target area A (an area where image processing is not performed), the control unit 130 P Therefore, when the object identification information acquisition unit 110 receives a position designation input (S110: YES), the control unit 130 determines that the position designated by the position designation input is included in the processing target area A. P It is further determined whether or not the point is outside the area (S112).

[0048] The position specified by the position input is the processing target area A. P If the specified position is outside the processing target area A (S112: YES), the control unit 130 determines whether or not the specified position is outside the processing target area A for at least one of the mechanisms for controlling the imaging range of the imaging device 20. P The control unit 130 calculates a control amount necessary to include the object in the processing target area A (S114). Then, the control unit 130 transmits a control signal based on the calculated control amount to the mechanism to be operated (S116). The mechanism operates in response to this control signal, and the position specified by the position specification input (the object selected as the tracking target that exists at that position) is included in the processing target area A. P Then, the object tracking device 10 detects a new image (the object selected as the tracking target is included in the processing target area A) generated by the imaging device 20 after the mechanism operates in response to the control signal. P (S118). Then, the object identification information acquisition unit 110 acquires object identification information for identifying an object selected as a tracking target, using information on the position designated by the position designation input (S120). The object identification information acquisition unit 110 first identifies a position on a new image that corresponds to the position designated in the image acquired in the process of S102. For example, the object identification information acquisition unit 110 can identify a position on the new image that corresponds to the designated position, based on the position designated on the image acquired in the process of S102 and the control amount (movement amount of the imaging range) of S114. Then, the object identification information acquisition unit 110 acquires a feature amount of an object detected at the position specified on the new image as object identification information for identifying an object to be tracked. Then, the object identification information acquired here is used to execute an object tracking process as illustrated in FIG. 7.

[0049] On the other hand, the position specified by the position input is the processing target area A PIf the object is located inside the area (S112: NO), the above-mentioned processes of S114 to S118 are not executed. In this case, the object identification information acquisition unit 110 uses information on the position specified by the position specification input to acquire object identification information for identifying the object to be tracked, which is a feature amount of the object detected at the position (S120). Then, the object tracking process as shown in FIG. 7 is executed using the object identification information acquired here.

[0050] If the object identification information acquisition unit 110 does not accept the position designation input in the judgment of S110 (S110: NO), the object identification information acquisition unit 110 further judges whether or not an object matching a predetermined condition has been detected in the object detection process of S104 (S122). Here, the predetermined condition is a condition for determining that the object is a tracking target object. Specific examples of the "predetermined condition" include, but are not limited to, "a person carrying a backpack," "a red sedan-type car," and "a two-seater bicycle." In addition, the "predetermined condition" may indicate characteristics related to the appearance of the suspect (hairstyle, clothing, belongings, etc.). Information related to such predetermined conditions may be stored in advance in the memory 1030 or the storage device 1040, or may be input by the user to the object tracking device 10 via the user terminal 30.

[0051] If an object that meets the predetermined conditions is detected in the object detection process of S104 (S122: YES), the object identification information acquisition unit 110 acquires the feature amount of the detected object as object identification information (S124). Then, using the object identification information acquired here, an object tracking process is performed as illustrated in Fig. 7. On the other hand, if an object that meets the predetermined conditions is not detected in the object detection process of S104 (S122: NO), the process returns to S102, and the above-mentioned process is repeated for an image newly acquired from the imaging device 20.

[0052] <<Object tracking processing>> The image processing unit 120 identifies an object corresponding to the object identification information acquired in the previous process as an object to be tracked from among the objects detected in the object detection process of S104, and acquires the position (position in the image coordinate system) of the object to be tracked (S126). Then, the image processing unit 120 outputs information indicating the position of the object to be tracked to the display of the user terminal 30 (S128), as shown in FIG. 10, for example. FIG. 10 is a diagram showing an example of the output of the image processing unit 120 based on the result of the object detection process. Note that in FIG. 10, the object identified by the object identification information is a vehicle V. 1 In this case, the image processing unit 120 detects the vehicle V 1 The information indicating the position of the object to be tracked is superimposed on the image acquired from the imaging device 20. For example, the image processing unit 120 may display the vehicle V 1 By superimposing information on the image that highlights the area corresponding to the vehicle V 1 The image processing unit 120 notifies the user that the vehicle V is being tracked. 1 (the object to be tracked) 1 The display color or state of the vehicle V 2 (Other detected objects) corresponding to frame F 2 The image processing unit 120 may be configured to be different from the vehicle V. 1 A message or marking indicating that the vehicle is being tracked may be displayed on the vehicle. 1 may be displayed in association with

[0053] Then, the control unit 130 determines whether or not it is necessary to control the imaging range of the imaging device 20 based on the position of the object to be tracked acquired in the process of S126 (S130). Specifically, the control unit 130 compares the position of the object to be tracked with the position of a predetermined area a (an area including the center of the image) as exemplified in FIG. 3, and determines whether or not it is necessary to control the imaging range of the imaging device 20.

[0054] When it is determined that the imaging range of the imaging device 20 needs to be controlled (S130: YES), for example, when the position of the tracked object is outside the predetermined area a, the control unit 130 controls at least one of the mechanisms for controlling the imaging range of the imaging device 20 to control the position of the tracked object within the processing target area A. P The control unit 130 calculates a control amount necessary to include the object to be tracked in the predetermined area a (S132). Then, the control unit 130 transmits a control signal based on the calculated control amount to the mechanism to be operated (S134). The mechanism operates in response to this control signal, so that the object to be tracked is included in the predetermined area a. On the other hand, when it is determined that control of the imaging range of the imaging device 20 is not necessary (S130: NO), the control unit 130 does not execute the above-mentioned process.

[0055] Then, the control unit 130 performs a determination as to whether or not to end the object tracking process (S136). Specific examples of the end conditions of the object tracking process include, but are not limited to, "an instruction to end the process has been input in the user terminal 30," "the movement limit of the mechanism that controls the imaging range of the imaging device 20 has been reached," and "the object tracking process has continued for a predetermined time or distance or longer." If the end conditions of the object tracking process are not met (S136: NO), the process returns to S126, and the object tracking process is continued using a new image acquired from the imaging device 20.

[0056] <<What to do when the tracked object is lost>> Here, when the tracked object moves suddenly, or when the tracked object approaches (passes) another object similar to the tracked object, the object tracking device 10 may lose sight of the tracked object. Here, "losing sight" means that the object tracking device 10 is no longer able to identify which object corresponds to the object identification information.

[0057] In this case, the object tracking device 10 executes the process shown in Fig. 11. Fig. 11 is a flowchart illustrating the flow of the process executed by the object tracking device 10 when the object to be tracked is lost.

[0058] The image processing unit 120 processes the area AP If the object to be tracked fails to be detected (the object to be tracked is lost) (S202: YES), the object identification information acquisition unit 110 acquires information indicating the position of the object in the image acquired from the imaging device 20 (hereinafter, also referred to as "object position information") (S204). When the object identification information acquisition unit 110 acquires the object position information, the control unit 130 determines whether the position indicated by the object position information is within the processing target area A. P It is further determined whether or not the point is outside (S206).

[0059] The position indicated by the object position information is the processing target area A P If it is outside the processing target area A (S206: YES), the control unit 130 determines that it is necessary to control the imaging range of the imaging device 20. In this case, the control unit 130 determines that, for at least one of the mechanisms for controlling the imaging range of the imaging device 20, the position indicated by the object position information is included in the processing target area A. P The control unit 130 calculates a control amount necessary to include the object in the processing target area A (S208). Then, the control unit 130 transmits a control signal based on the calculated control amount to the mechanism to be operated (S210). The mechanism operates in response to the control signal, and the position indicated by the object position information (the lost object present at that position) is included in the processing target area A. P Then, the object tracking device 10 detects a new image (where the lost object is included in the processing target area A) generated by the imaging device 20 after the mechanism operates in response to the control signal. P The object identification information acquisition unit 110 acquires an image of the object (an image of the object in a state of being included in the image) (S212). Then, the object identification information acquisition unit 110 acquires object identification information for re-identifying the lost object by using the position indicated by the object position information (S214). Then, the object tracking process as shown in FIG. 7 is resumed by using the object identification information acquired here.

[0060] On the other hand, the position indicated by the object position information is the processing target area A PIf the object is within the range (S206: NO), the above-mentioned processes of S206 to S212 are not executed. In this case, the object identification information acquisition unit 110 acquires object identification information for re-identifying the lost object by using the position indicated by the object position information (S214). Then, the object tracking process as shown in FIG. 7 is executed by using the object identification information acquired here.

[0061] The process flow from S206 to S214 in FIG. 11 described above is similar to the process flow from S114 to S120 in FIG.

[0062] The above-mentioned process will be specifically described with reference to Figs. 12 and 13. Figs. 12 and 13 are diagrams illustrating an example of a flow of acquiring object position information when a tracking target object is lost. For example, when a tracking target object (vehicle V 1 ) moves at a speed exceeding the operating speed of the mechanism that controls the imaging range of the imaging device 20, the object to be tracked (vehicle V 1 ) is the processing area A P In this case, the image processing unit 120 cannot detect the object to be tracked and loses sight of the object. However, as shown in FIG. 12, E Regarding the processing area A P The tracked object (vehicle V 1 In this case, the processing area A P When the user selects the position of the object to be tracked in the area outside the area, the current position of the lost object can be notified to the object tracking device 10. The object identification information acquisition unit 110 acquires information on the position selected by the user in this manner (for example, the position indicated by the pointer P) as the above-mentioned object position information.

[0063] In the example of FIG. 12, the object position information obtained is the processing object area A P13, the control unit 130 generates and transmits a control signal for operating a mechanism for controlling the imaging range of the imaging device 20 based on the object position information acquired by the object identification information acquisition unit 110. This control signal causes the position selected by the user (for example, the position indicated by the pointer P) to be included in the processing target area A. P 13, the control unit 130 transmits a control signal to control the operation of the mechanism for adjusting the zoom ratio, orientation, position, etc. of the imaging device 20 so that the imaging range of the imaging device 20 moves in a lower left direction. Then, as shown in FIG. 13, the imaging range of the imaging device 20 moves in a lower left direction. The image processing unit 120 calculates the processing target area A in the imaging range moved by the control signal. P By performing image processing on the target object (vehicle V 1 ) can be resumed.

[0064] [Second embodiment] The object tracking device 10 of this embodiment differs from the first embodiment described above in that it further includes a recording unit as described below.

[0065] <Example of functional configuration> Fig. 14 is a diagram showing an example of a functional configuration of the object tracking device 10 in the second embodiment. As shown in Fig. 14, the object tracking device 10 of this embodiment includes a recording unit 122 in addition to the configuration described in the first embodiment. After detecting a tracking target object, the recording unit 122 stores images acquired from the imaging device 20 as recorded data in a predetermined storage area (for example, the storage device 1040, etc.) until a predetermined end condition is satisfied.

[0066] <Processing flow> Hereinafter, the processing executed by the object tracking device 10 of the present embodiment will be described with reference to the drawings. Fig. 15 is a flowchart illustrating the flow of the recording processing executed by the object tracking device 10 of the second embodiment.

[0067] First, the recording unit 122 determines whether or not an object tracking process has been started (S302). Specifically, the recording unit 122 determines whether or not the object identification information has been acquired by the object identification information acquisition unit 110 in the process illustrated in Fig. 6. Here, if the object tracking process has not been started (S302: NO), the process described below is not executed.

[0068] When the object tracking process is started, that is, when the object identification information acquisition unit 110 acquires the object identification information (S302: YES), the recording unit 122 buffers the image (video data) acquired from the imaging device 20, for example, in the memory 1030 (S304). After that, the recording unit 122 continues to buffer the image (video data) acquired from the imaging device 20 until the object tracking process is completed (S306: NO). Then, when the object tracking process is completed (S306: YES), the recording unit 122 stores the video data buffered in the memory 1030, etc., in the storage device 1040, etc., as recording data indicating the result of the object tracking process (S308).

[0069] As described above, in this embodiment, when the object tracking process described in the first embodiment is started, based on the video data acquired from the imaging device 20, recorded data showing the result of the object tracking process is generated and stored. This recorded data showing the result of the past object tracking process is useful when the user performs a confirmation task. For example, the user can carefully check the movement of the tracked object by reviewing the recorded data about the tracked object of interest.

[0070] Here, in a situation where an object tracking process is not being performed (i.e., target identification information is not acquired), the recording unit 122 may generate and store a time lapse video using, for example, images (video data) acquired from the imaging device 20. By checking the time lapse video, the user can roughly grasp the situation before and after the tracking process of a certain object is started. In this case, the recording unit 122 may store the above-mentioned recording data and the time lapse video as separate files in the storage device 1040 or the like, or may store them as a single file in the storage device 1040 or the like.

[0071] [Third embodiment] The object tracking device 10 of this embodiment differs from the first embodiment in that it further includes a report generating unit as described below.

[0072] <Example of functional configuration> 16 is a block diagram showing an example of a functional configuration of the object tracking device 10 in the third embodiment. As shown in FIG. 16, the object tracking device 10 of the present embodiment further includes a report generating unit 140.

[0073] The report generating unit 140 generates a report file showing the result of the object tracking process. The report file generated by the report generating unit 140 includes, for example, an image showing the object to be tracked, information on the period (date and time) during which the tracking process was performed, information on the location where the tracking process was performed, and the movement trajectory of the object to be tracked. In addition, when the object tracking device 10 includes the recording unit 122 described in the second embodiment, the report file may further include a link address to the recorded data generated according to the object tracking process. The image showing the object to be tracked is, for example, a representative image among the images acquired from the imaging device 20 (for example, when the recorded data is generated in the above-mentioned second embodiment, a thumbnail image of the recorded data), etc. The report generating unit 140 may further include text information related to the object to be tracked in the report file in addition to the image. The text information related to the object to be tracked is, for example, information showing the category of the object (person, vehicle, etc.) and the attributes of the object (for example, "carrying a backpack" for a person, and "red sedan type" for a vehicle). The information regarding the location where the tracking process was performed is, for example, information indicating the location (such as an address) where the imaging device 20 is installed, or information identifying an individual imaging device 20, such as a number uniquely assigned to each imaging device 20.

[0074] <Processing flow> Hereinafter, the processing executed by the object tracking device 10 of the present embodiment will be described with reference to the drawings. Fig. 17 is a flowchart illustrating the flow of a report generation processing executed by the object tracking device 10 of the third embodiment.

[0075] First, the report generating unit 140 judges whether or not the object tracking process has started (S402). Specifically, the report generating unit 140 judges whether or not the object identifying information has been acquired by the object identifying information acquiring unit 110 in the process as exemplified in Fig. 6. Here, if the object tracking process has not started (S402: NO), the process described below is not executed.

[0076] When the object tracking process is started, i.e., when the object identification information acquisition unit 110 has acquired object identification information (S402: YES), the report generation unit 140 collects information to be included in the report file from the start to the end of the object tracking process (S404).

[0077] For example, the report generating unit 140 determines one representative image from among images acquired from the imaging device 20 while the object tracking process is being performed. The report generating unit 140 can digitize the visibility of the object for each image based on, for example, the orientation, the degree of blur, the brightness, etc. of the object to be tracked, which is the subject, and can include the optimal image based on the digitized value as a representative image in the report file. The report generating unit 140 can also include the result of inputting the image of the imaging device 20 to the object detector (for example, "vehicle" or "red sedan type vehicle") as text information in the report file. The report generating unit 140 can also acquire the start time and end time of the object tracking process, and can include them in the report file as information indicating the period during which the object tracking process was performed. The report generating unit 140 can also acquire the installation location of the imaging device 20 that generated the image used in the object tracking process, a number unique to the imaging device 20, etc., and can include them in the report file as information indicating the point at which the object tracking process was performed. Furthermore, the report generating unit 140 can include in the report file information indicating the movement trajectory of the object, which is obtained by converting the position of the object to be tracked in each image into a position on a map. In this case, the report generating unit 140 can use, for example, a function that converts the position of the image into a position on a map based on the attitude (pan / tilt angle) of the imaging device 20 and the position of the imaging device 20. Such a function is stored in advance in, for example, the memory 1030 or the storage device 1040. Furthermore, in the case where the object tracking device 10 further includes the recording unit 122 described in the second embodiment, the report generating unit 140 can include in the report file information indicating a link address of the recording data output at the end of the object tracking process.

[0078] Then, the report generation unit 140 outputs the report file generated based on the information collected in S404 to the storage device 1040, the user terminal 30, or the like (S406).

[0079] As described above, in this embodiment, when an object tracking process is executed, a report file about the object tracking process is generated and output. According to the configuration of this embodiment, the user can easily manage and understand the results of the object tracking process.

[0080] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.

[0081] In addition, in the sequence diagrams and flow charts used in the above description, multiple steps (processes) are described in order, but the order of execution of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the figures can be changed to the extent that the content is not affected. In addition, the above-mentioned embodiments can be combined to the extent that the content is not contradictory.

[0082] A part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. 1. An object identification information acquisition means for acquiring object identification information for identifying an object to be tracked; an image processing means for detecting an object to be tracked by performing image processing on a processing target area, which is a partial area of ​​an image acquired from an imaging device, and tracking the detected object; a control means for controlling an imaging range of the imaging device so that the object to be tracked is included in the processing target area; An object tracking device comprising: 2. the object identification information acquisition means acquires the object identification information in response to an input of a position of the tracked object in the image; 1. An object tracking device as described in claim 1. 3. the object identification information acquisition means acquires the object identification information in response to an input for selecting a position of the object to be tracked on the image displayed on a display. 2. An object tracking device as described above. 4. the target identification information acquisition means executes a process of acquiring information indicating a position of the tracked object in the image when the image processing fails to detect the tracked object. An object tracking device according to any one of 1 to 3. 5. The processing target area is an area including a central portion of the image. 5. An object tracking device according to any one of 1 to 4. 6. the control means controls an imaging range of the imaging device so that the object to be tracked is included in a predetermined area that includes a central portion of the image and is a part of the processing target area. 5. An object tracking device as described above. 7. The size of the processing target area is predetermined. 1. An object tracking device according to any one of claims 1 to 6. 8. the size of the processing target area is determined based on the size of resources that can be allocated to image processing of the processing target area. 7. An object tracking device as described in claim 7. 9. The tracking device further includes a recording unit configured to store, as recorded data, images acquired from the imaging device during a period from when the tracking target object is detected until a predetermined end condition is satisfied. 9. An object tracking device according to any one of 1 to 8. 10. the image processing means displays information indicating the processing target area on a display by superimposing the information on the image. 1. An object tracking device according to any one of claims 1 to 9. 11. the control means controls an imaging range of the imaging device by operating at least one of a mechanism for controlling the zoom of the imaging device, a mechanism for controlling the orientation of the imaging device, and a mechanism for controlling the position of the imaging device. An object tracking device according to any one of 1 to 10. 12. The computer Acquire target identification information that identifies the object to be tracked; performing image processing on a processing target area, which is a partial area of ​​an image acquired from an imaging device, to detect an object to be tracked and track the detected object; controlling an imaging range of the imaging device so that the object to be tracked is included in the processing target area; The object tracking method includes: 13. The computer, acquiring the target identification information in response to an input of a position of the tracked target in the image; 13. The object tracking method according to claim 12, further comprising: 14. The computer, acquiring the target object identification information in response to an input for selecting a position of the tracked object on the image displayed on a display; 14. The object tracking method according to claim 13, further comprising: 15. The computer, executing a process of acquiring information indicating a position of the tracked object in the image when the image processing fails to detect the tracked object; 15. The object tracking method according to any one of 12. to 14., 16. The processing target area is an area including a central portion of the image. 16. An object tracking method according to any one of claims 12 to 15. 17. The computer, controlling an imaging range of the imaging device so that the object to be tracked is included in a predetermined area that includes a central portion of the image and is a part of the processing target area; 16. The object tracking method according to claim 15, further comprising: 18. The size of the processing target area is predetermined. 18. An object tracking method according to any one of claims 12 to 17. 19. the size of the processing target area is determined based on the size of resources that can be allocated to image processing of the processing target area. 18. The object tracking method according to claim 18. 20. The computer, storing, as recorded data, images acquired from the imaging device during the period from when the object to be tracked is detected until a predetermined termination condition is satisfied; 20. The object tracking method according to any one of claims 12 to 19, further comprising: twenty one. The computer, and displaying information indicating the processing target area on a display by superimposing the information on the image. 21. The object tracking method according to any one of 12. to 20., twenty two. The computer, controlling an imaging range of the imaging device by operating at least one of a mechanism for controlling the zoom of the imaging device, a mechanism for controlling the orientation of the imaging device, and a mechanism for controlling the position of the imaging device; 22. The object tracking method according to any one of claims 12 to 21. twenty three. A program for causing a computer to execute the object tracking method according to any one of 12. to 22.

Claims

1. At least one computer determining a size of a processing target area in which a process for detecting a person in an image is to be executed based on a size of an excess computing resource, and accepting information including the determined size of the processing target area as a setting of the processing target area; Detecting a person appearing within the processing target area; A person captured within the processing target area is set as a tracking target; acquiring position coordinates of a predetermined area, which is a part of the processing target area and in which the tracking target is to be displayed; controlling an imaging range of a camera based on a position coordinate of the predetermined area in which the tracking target is to be displayed, so that the position of the tracking target is included within the range of the predetermined area; If the position of the tracking target is included within the range of the predetermined area, the tracking target is continued to be tracked without controlling the imaging range of the camera. Methods for tracking objects.

2. The at least one computer a display device that displays the image and information indicating the processing target region; The method of claim 1 , further comprising:

3. The at least one computer Controlling the zoom ratio of the camera; The object tracking method according to claim 1 or 2, comprising:

4. The at least one computer Controlling the orientation of the camera; The object tracking method according to claim 1 , further comprising:

5. The position coordinates are position coordinates on the image displayed on a display device. The object tracking method according to any one of claims 1 to 4.

6. The at least one computer accepting a position designation input for designating a position on the image; setting the tracking target based on the position designation input; The object tracking method according to claim 1 , further comprising:

7. The at least one computer: a display device that displays the image displays a frame indicating the tracking target together with the image; The method of claim 1 , further comprising:

8. The at least one computer: Accepting input defining the processing target area; The method of claim 1 , further comprising:

9. The processing target area is set to a range in which a person is detected from the image in a horizontal direction and a range in which a person is detected from the image in a vertical direction. The method for tracking an object according to any one of claims 1 to 8.

10. On the computer, determining a size of a processing target area in which processing for detecting a person in an image is to be performed based on a size of an excess computing resource, and accepting information including the determined size of the processing target area as a setting of the processing target area; A process of detecting a person appearing within the processing target area; A process of setting one person appearing within the processing target area as a tracking target; A process of acquiring position coordinates of a predetermined area, which is a part of the processing target area, in which the tracking target is to be displayed; a process of controlling an imaging range of a camera based on position coordinates of the predetermined area in which the tracking target is displayed so that the position of the tracking target is included within the range of the predetermined area; If the position of the tracking target is included within the range of the predetermined area, a process of continuing tracking of the tracking target without controlling an imaging range of the camera; A program that executes the following.

11. The computer includes: a display device that displays the image further executes a process of displaying information indicating the processing target region together with the image; The program according to claim 10.

12. The computer includes: and further executing a process of controlling a zoom ratio of the camera. The program according to claim 10 or 11.

13. The computer includes: and further executing a process of controlling the orientation of the camera. The program according to any one of claims 10 to 12.

14. The position coordinates are position coordinates on the image displayed on a display device. The program according to any one of claims 10 to 13.

15. The computer includes: A process of receiving a position designation input for designating a position on the image; and further executing a process of setting the tracking target based on the position designation input. The program according to any one of claims 10 to 14.

16. The computer includes: a display device that displays the image further executes a process of displaying a frame indicating the tracking target together with the image; The program according to any one of claims 10 to 15.

17. The computer includes: further executing a process of receiving an input defining the processing target region; The program according to any one of claims 10 to 16.

18. The processing target area is set to a range in which a person is detected from the image in a horizontal direction and a range in which a person is detected from the image in a vertical direction. The program according to any one of claims 10 to 17.

19. a target area setting means for determining a size of a processing target area in which a process for detecting a person in an image is executed based on a size of an excess computing resource, and for receiving information including the determined size of the processing target area as a setting of the processing target area; a person detection means for detecting a person appearing within the processing target area; a tracking target detection means for setting a person appearing within the processing target area as a tracking target; an information acquisition means for acquiring position coordinates of a predetermined area, which is a part of the processing target area and in which the tracking target is to be displayed; a camera control means for controlling an imaging range of a camera based on position coordinates of the predetermined area in which the tracking target is displayed so that the position of the tracking target is included within the range of the predetermined area; Equipped with when the position of the tracking target is included within the range of the predetermined area, the camera control means continues tracking of the tracking target without controlling the imaging range of the camera. Object tracking device.

Images