Display control device and method for controlling the same, and imaging system

Abstract

To allow a user to grasp a relationship between images photographed by different cameras.SOLUTION: A display control device performs display control of a photographed image acquired by an imaging apparatus having a first imaging unit that photographs a predetermined area, and a second imaging unit that can photograph an area range including at least the predetermined area by performing control to change a photographing direction. The display control device has: image acquisition means that acquires a first photographed image obtained by the first imaging unit, and a second photographed image obtained by the second imaging unit; and display control means that displays, on a display unit, the first photographed image and area display corresponding to a photographing area of the second imaging unit. When the second photographed image is an image obtained by photographing the outside of the predetermined area, the display control means displays the area display corresponding to the photographing area of the second imaging unit in an area outside an area for displaying the first photographed image on the display unit.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to display control of images captured by a plurality of cameras. [Background technology]

[0002] Network cameras that can transmit captured video and audio via a network are widely used. Network cameras include wide-area cameras that use wide-angle lenses to capture a wide range in a fixed direction, and PTZ cameras that use the pan-tilt-zoom (PTZ) function to capture a narrow range with a variable capture direction.

[0003] A system that combines a wide-area camera and a PTZ camera is also known. Patent Document 1 discloses a method for controlling the shooting direction of a PTZ camera by specifying an outer area based on the shooting range of the wide-area camera when shooting an area outside the shooting range of the wide-area camera with the PTZ camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-186635 A Summary of the Invention [Problem to be solved by the invention]

[0005] In a system combining a wide-area camera and a PTZ camera, if the shooting range of the PTZ camera is wider than the shooting range of the wide-area camera, the PTZ camera may recognize an object outside the shooting range of the wide-area camera. In that case, the user cannot understand the positional relationship in which the object was recognized with respect to the shooting range of the wide-area camera. The above-mentioned Patent Document 1 specifies the shooting direction, and cannot deal with such a problem.

[0006] The present invention has been made in consideration of such problems, and has an object to provide a technique that enables a user to easily grasp the relationship between images captured by different cameras. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the display control device according to the present invention has the following configuration. That is, the display control device controls the display of a captured image acquired by an imaging device having a first imaging unit that captures a predetermined area and a second imaging unit that can capture an area range that includes at least the predetermined area by changing and controlling the imaging direction, an image acquisition means for acquiring a first captured image acquired by the first imaging section and a second captured image acquired by the second imaging section; a display control means for displaying, on a display unit, the first captured image and an area display corresponding to the capture area of ​​the second imaging unit; having When the second captured image is an image captured outside the specified area, the display control means displays the area display corresponding to the capturing area of ​​the second imaging unit in an area outside the area displaying the first captured image on the display unit. Effect of the Invention

[0008] According to the present invention, it is possible to provide a technique that enables a user to easily understand the relationship between images captured by different cameras. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an overall configuration of an imaging system. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the imaging apparatus. [Diagram 3] FIG. 2 is a block diagram showing a hardware configuration of a terminal. [Figure 4] FIG. 4 is a diagram showing an example of a screen displayed on a display device. [Diagram 5]FIG. 13 is a diagram showing an example of a screen showing a recognition processing result. [Figure 6] FIG. 11 is a diagram illustrating a display form of a recognition processing result. [Figure 7] 4 is a flowchart illustrating an operation of the imaging system. [Figure 8] FIG. 11 is a diagram showing an example of a screen showing a recognition processing result (second embodiment). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] (First embodiment) As a first embodiment of a display control device according to the present invention, an imaging system for monitoring vehicles in a parking lot will be described below as an example.

[0012] <Device configuration> Fig. 1 is a diagram showing the overall configuration of an imaging system. The imaging system is composed of an imaging device 100, a terminal 200, and a network (IP network) 300 connecting them. The imaging device 100 is a surveillance camera installed on a ceiling, a utility pole, or the like. The terminal 200 is a device that displays images and various information acquired from the imaging device 100 and allows a user to remotely operate the imaging device 100. Note that, although Fig. 1 illustrates one imaging device 100 and one terminal 200, there is no particular limit to the number.

[0013] The imaging device 100 accommodates a wide-area camera 110 and a pan-tilt-zoom (PTZ) camera 120. The wide-area camera 110 is a camera that uses a wide-angle lens or the like to capture a given area with a relatively wide angle of view (for example, a 100-degree angle of view). The PTZ camera 120 is a camera that captures an image with a relatively narrow angle of view (for example, less than 30 degrees) and is capable of changing the image capture direction and optical zoom magnification. The PTZ camera 120 is configured to be capable of capturing an area range that includes the capture range of the wide-area camera 110 by controlling the change of the capture direction. In the following description, it is assumed that the wide-area camera 110 is a camera with a fixed angle of view, but it may be a camera that is capable of zooming. In addition, the PTZ camera 120 is capable of capturing an image outside the capture range of the wide-area camera 110 by changing the capture direction (performing pan and tilt operations).

[0014] Furthermore, the imaging device 100 has a function of transmitting images (frame images) obtained by the wide area camera 110 and the PTZ camera 120 to an external device via the network 300. The terminal 200, which is an external device, receives images from the imaging device 100 via the network and displays them on a display unit. The terminal 200 also has a function of issuing various commands to the imaging device 100.

[0015] 2 is a block diagram showing the functional configuration of the imaging device 100. The imaging device 100 has a wide area camera 110, a PTZ camera 120, a control unit 130 that controls these and processes the captured images obtained, a storage unit 140, and a communication unit 150 that communicates with an IP network 300. The imaging frame rates of the wide area camera 110 and the PTZ camera 120 are not particularly important, but are assumed to be 15 frames / second here. The imaging device 100 has two imaging modes (remote control mode and monitoring mode). The imaging mode is switched according to a command transmitted from the terminal 200.

[0016] In the remote control mode, the control unit 130 performs pan, tilt, and zoom control of the PTZ camera 120 in accordance with a request command from the terminal 200, and transmits an image obtained by the PTZ camera 120 to the terminal 200. At this time, a wide-range image (described later) obtained by the wide-area camera 110 is also transmitted to the terminal 200.

[0017] In the monitoring mode, the control unit 130 autonomously controls the pan, tilt, and zoom of the PTZ camera 120. For example, a tracking operation is performed to control the PTZ camera 120 so that the object is at the center of the image according to the movement of the detected object. For example, the captured moving images are compared between frames to detect an area with a change (such as movement) (moving object detection function), and the shooting direction of the camera is controlled so that the object is located at the center of the shooting range (tracking function). Furthermore, a recognition operation is performed by comparing the information stored as a database with the result of analyzing the characteristic points of the detected object (recognition function). Then, the control unit 130 transmits the image obtained by the PTZ camera 120 and the image obtained by the wide area camera 110 to the terminal 200. Note that the following description will mainly describe the operation when the device is operating in the monitoring mode.

[0018] The storage unit 140 is a non-volatile memory such as a flash memory or a hard disk. The storage unit 140 stores model information of a "vehicle" for detecting a "vehicle" as a subject from a moving object, and feature information for recognizing (identifying) a vehicle (specific vehicle) to be searched. There is no particular limit to the number of vehicles to be searched. The model information and feature information of the vehicle are received accompanying a predetermined registration command from the terminal 200. When the registration command is received, the accompanying received information is stored in the storage unit 140. The subject is not limited to a vehicle, but is intended to be a subject having a specific feature such as a "human face" or a "human body". Note that the detection of the vehicle and the recognition of the specific vehicle themselves are performed using well-known technology, and detailed description thereof will be omitted here.

[0019] The communication unit 150 is an interface for connecting to the IP network 300. The connection between the communication unit 150 and the IP network 300 may be wired or wireless.

[0020] The wide-angle camera 110 has an imaging optical system 111, an imaging element 112, and an image processing unit 113. The imaging optical system 111 is composed of a wide-angle lens for acquiring an image with a wide angle of view. The imaging element 112 is composed of a two-dimensional image sensor such as a CCD or CMOS, and supplies an electric signal obtained by photoelectric conversion to the image processing unit 113. The image processing unit 113 converts the electric signal received from the imaging unit 112 into digital data and performs processes to improve image quality, including demosaic processing. In addition, it performs gain processing to increase the amount of acquired light as a signal level, and further performs processes such as outputting a reset signal to the imaging element 112 to clear the accumulated charge. Furthermore, the image processing unit 113 performs exposure control by adjusting the amount of light incident on the imaging element by changing the settings of the shutter speed, the aperture value, and the gain processing. Then, the image processing unit 113 supplies the image data after the above processes to the control unit 130.

[0021] The PTZ camera 120 has an imaging optical system 121, an imaging element 122, an image processing unit 123, and a driving unit 124. The imaging optical system 121 is composed of a zoom lens, a focus lens, an aperture mechanism, etc. The processing contents of the imaging element 122 and the image processing unit 123 are similar to those of the imaging element 112 and the image processing unit 113, respectively. The driving unit 124 drives various motors (not shown) for controlling panning, tilting, and zoom lens operation according to control signals from the control unit 130. As a result of driving by this driving unit 124, the shooting direction (optical axis direction) and the optical zoom magnification of the PTZ camera 120 can be changed.

[0022] The control unit 130 has a drive control unit 131, a camera determination unit 132, a subject detection unit 133, a recognition processing unit 134, an encoding unit 135, a communication control unit 136, and a command interpretation unit 137. The control unit 130 may be composed of, for example, a central processing unit (CPU), a ROM that stores programs executed by the CPU, and a RAM used as a work area for the CPU. In this case, each processing unit in the control unit 130 is realized in software by the CPU executing a program. However, some or all of the processing units may be realized by dedicated hardware.

[0023] The drive control unit 131 performs processing for generating drive signals for controlling the pan and tilt operations of the drive unit 124 of the PTZ camera 120 and the zoom magnification. The drive unit 124 drives various motors for PTZ operations based on these drive signals. The drive control unit 131 also controls focusing, aperture, and the like by the wide area camera 110 and the PTZ camera 120. The drive control unit 131 has a tracking function, and adjusts the pan, tilt, and zoom amounts according to the amount of movement of the subject detected by a subject detection unit 133 described below.

[0024] The camera determination unit 132 determines whether the acquired video, detected subject information described below, and result information of the recognition process are from the wide area camera 110 or the PTZ camera 120, and performs processing on each camera according to the determination result.

[0025] The subject detection unit 133 detects a subject by comparing the images (frame images) acquired from the wide area camera 110 and the PTZ camera 120 between frames (i.e., in the time series direction), and judging that it is a moving object when there is a difference between the frames. The size information of the subject detected from the difference information (the size of a circumscribing rectangle that contains the moving object), feature information, and position information are transmitted to the recognition processing unit 134.

[0026] The recognition processing unit 134 first identifies the subject from the information from the subject detection unit 133, and determines, for example, whether the detected subject is a vehicle or a person. Furthermore, when one or more new subjects are detected by the subject detection unit 133, the recognition processing unit 134 selects one of them.

[0027] Here, when the wide-area camera 110 and the PTZ camera 120 detect a new object, each may perform the recognition process independently, but the wide-area camera 110 and the PTZ camera 120 may cooperate to perform the recognition process. For example, the object detected by the wide-area camera 110 may be photographed in detail by the PTZ camera 120. In this case, the recognition processing unit 134 controls the drive control unit 131 to align the optical axis direction of the PTZ camera 120 with the direction of the object detected by the wide-area camera 110 and control the zoom magnification to obtain a high-precision image of the selected object. Then, the recognition processing unit 134 extracts a feature amount of the object from the image obtained by the PTZ camera 120. The degree of match between the extracted feature amount and the feature amount for recognizing a search object (for example, a vehicle) stored in the storage unit 140 is obtained, a recognition process is performed to determine whether the selected object is a search object (whether it is the car being searched for) and the recognition result is output.

[0028] For example, in license plate recognition processing, the license plate area is extracted from the captured image, image processing (contrast adjustment, binarization, etc.) is performed, and characters are identified based on the outline and size of the characters. There are known methods for identifying these characters using a large amount of learning data.

[0029] The encoding unit 135 encodes the image data obtained by the wide area camera 110 and the PTZ camera 120 from the image processing units 113 and 123 to generate encoded data. The encoding unit 135 also generates information related to the video as metadata. For example, for the image data obtained by the PTZ camera 120, information such as pan, tilt, and zoom information and the coordinates of the subject is generated. The communication control unit 136 then transmits the encoded data generated by the encoding unit 135 to the terminal 200 via the communication unit 150. The communication control unit 136 can also lay out the image obtained from the wide area camera 110 and the video obtained from the PTZ camera 120 in one video and transmit the laid out video. Furthermore, when the communication control unit 136 receives various request commands from the terminal 200 via the communication unit 150, it notifies the command interpretation unit 137 of the request command.

[0030] The command interpretation unit 137 analyzes the request command notified by the communication control unit 136 and executes processing according to the request command. For example, when a mode transition command is received, the command interpretation unit 137 executes processing to transition to a specified mode.

[0031] In the remote control mode, the PTZ camera 120 is controlled according to the pan angle, tilt angle, and zoom magnification in response to a request command from the terminal. Then, an image obtained by the PTZ camera 120 is transmitted to the terminal 200 together with a wide-range image obtained by the wide-area camera 110. On the other hand, in the monitoring mode, the monitoring mode processing described later is performed. In the monitoring mode, the command interpretation unit 137 processes a registration command for registering model information for subject detection and feature amounts of a search target subject in the storage unit 140.

[0032] FIG. 3 is a block diagram showing a hardware configuration of the terminal 200. The functional configuration related to display control, which will be described later, is also shown. The terminal 200 shows the configuration of an information processing device, typified by a personal computer (PC). The terminal 200 has a CPU 201 that controls the entire device, a ROM 202 that stores a BIOS and a boot program, and a RAM 203 that functions as a work area. The terminal 200 also has a HDD 204 that stores an operating system (OS), application programs, and the like, and a network interface (I / F) 205 for communicating with an IP network 300. The terminal 200 also has an operation unit 206 that is composed of a keyboard, mouse, touch panel, and the like for inputting instructions from a user. The terminal 200 further has an input / output I / F 207, a display device 208, and a display control unit 209.

[0033] The display control unit 209 has a determination unit 210, a frame generation unit 211, and a synthesis unit 212, which are functions for grasping the relationship between the wide-area camera 110 and the PTZ camera 120. In the above configuration, when the power supply of the terminal 200 is turned on, the CPU 201 loads the OS stored in the HDD 204 into the RAM 203 according to the boot program stored in the ROM 202 and executes it, thereby functioning as an information processing device. As a result, the operation unit 206 and the display device 208 function as a user interface. Then, when the user operates the operation unit 206 to instruct the execution of an application program, the terminal 200 loads the application stored in the HDD 204 into the RAM 203 and executes it. As a result, the terminal 200 functions as a device that controls the imaging device 100 and displays the captured image. The display control unit 209 performs an operation according to the operation unit 206, thereby realizing an operation intended by the user.

[0034] Note that each function of the display control unit 209 is realized by software (by the CPU 201 executing a program), but to make the operation easier to understand, it is described below separately from the CPU 201. Furthermore, the display control unit 209 also has a role of carrying out a frame generation function of the subject detection unit 133 of the imaging device 100, a subject identification function of the recognition processing unit 134, a metadata creation function of the encoding unit 135, and the like.

[0035] The determination unit 210 grasps the occurrence timing and completion timing of the recognition process and calculates the position information. The position information may be calculated as coordinates in the pan / tilt direction or as a geographical position (latitude / longitude, etc.). Then, it is determined whether the position is within the shooting range of the wide area camera 110 or outside the shooting range. For example, when the recognition process is performed on an object that is being tracked by the PTZ camera 120, the determination unit 210 calculates the position of the object at the time of recognition completion. Specifically, the determination unit 210 calculates the position of the object from the driving amount of the pan / tilt drive of the PTZ camera 120. For example, the imaging device 100 may be equipped with a global positioning system (GPS) receiver and stored as absolute value information, and the accuracy may be improved by performing calculations from the absolute value information. The calculated position information is stored as coordinate information in a storage device such as the HDD 204. For example, the encoding unit 135 adds the coordinate information stored in the HDD 204 to the video data as metadata, making it easier to handle the data.

[0036] The frame generating unit 211 generates a frame for information to be displayed on the display device 208. For example, the frame includes a layout frame when images captured by the wide area camera 110 and the PTZ camera 120 are displayed on the display device 208. In addition, the frame may include a setting area frame for accepting operations by the operation unit 206, and a display frame for displaying the determination result of the recognition process.

[0037] The synthesis unit 212 generates a synthetic image by superimposing the images captured by the wide area camera 110 and the PTZ camera 120 based on the frame generated by the determination unit 210 and the frame generation unit 211. For example, as described later with reference to Fig. 5, the image captured by the PTZ camera 120 is synthesized around the image captured by the wide area camera 110.

[0038] With the above-mentioned configuration, the display control unit 209 is responsible for exchanging data related to display on the display device 208. Furthermore, for example, when an image of a specific subject (a specific vehicle or a specific person's face) is input from the input / output I / F 207, a feature amount for recognizing the subject is extracted, and a command for registering the feature amount in the imaging device 100 is generated.

[0039] Furthermore, some of the functions of the control unit 130 in the imaging device 100 (for example, the subject detection unit 133 and the recognition processing unit 134) and the functions of the storage unit 140 can be performed by the CPU 201 and the HDD 204, respectively.

[0040] <Control of the screen displayed on the display device 208 of the terminal 200> 4 to 7, the display form of the recognition processing result displayed on the display device 208 of the terminal 200 will be described. As described above, in this embodiment, the PTZ camera 120 can also capture images outside the capture range of the wide area camera 110 by changing the capture direction (performing pan and tilt operations).

[0041] 4 is a diagram showing an example of a screen displayed on the display device 208. A display screen 401 shows the display screen of the display device 208 while an application is being executed. A display area 402, a display area 403, and an operation area 404 are arranged on the display screen 401. The display area 402 displays a wide-range image received from the wide-area camera 110. The display area 403 displays an image obtained by the PTZ camera 120. The operation area 404 has graphical user interface (GUI) components (buttons, sliders, etc.) related to user operations arranged therein.

[0042] Here, the imaging device 100 operating in the monitoring mode is shown to have detected a moving object 405 within a wide-range image displayed in a display area 402. The detected moving object 405 is then tracked and photographed by the PTZ camera 120, and the photographed image obtained by the PTZ camera 120 is displayed in a display area 403. An area 406 (or area 407) for displaying the result of authentication processing is configured on the outer periphery of the display area 402.

[0043] Fig. 4(a) shows a state in which an area 406 for displaying the result of the authentication process is provided around a display area 402 of an image captured by the wide area camera 110. On the other hand, Fig. 4(b) shows a state in which an area 407 indicating the image capturing range of the PTZ camera 120 is provided instead of the area 406.

[0044] In a conventional system in which a wide-area camera and a PTZ camera are combined, only the image capturing range of each camera is displayed. Therefore, when the PTZ camera captures an image outside the image capturing range of the wide-area camera, the user cannot grasp the relationship between the images captured by each camera (the positional relationship of the image capturing range). On the other hand, in this embodiment, the image capturing range of the PTZ camera 120 is displayed as the area 406 (or the area 407) in association with the positional relationship of the image captured by the wide-area camera 110. This makes it possible to express the relationship between the image capturing range of the wide-area camera 110 and the image capturing range of the PTZ camera 120, and the user can easily grasp such a relationship. In this embodiment, the image capturing range of the PTZ camera 120 is expressed by a frame, but the captured images obtained by the PTZ camera 120 may be synthesized and displayed as an expression that is close to the actual image capturing situation.

[0045] A button for mode selection (monitoring mode or remote control mode) is arranged in the operation area 404. Also arranged are GUI components for drive control in remote control mode (specifying pan angle, tilt angle, and zoom magnification) and GUI components for setting search conditions in monitoring mode.

[0046] In the operation area 404, furthermore, GUI components for setting the recognition process are arranged. For example, the recognition process to be performed is set for each of the wide-area camera 110 and the PTZ camera 120. For example, when setting the recognition process for license plates, a "vehicle" is selected as the detection target, and the detection area (area to be the target of the recognition process) and the like are set. When setting the recognition process for people, a "person" is selected as the detection target, and individual settings required for person recognition (such as setting the eye distance) are made.

[0047] 5A and 5B are diagrams showing examples of a screen showing the results of the recognition process, in which (a) and (b) show an example of an image displayed in the area 407 in FIG.

[0048] 5(a) is a diagram showing a state where the recognition process of the PTZ camera 120 is completed within the shooting range of the wide area camera 110. Specifically, after a moving object 405 is detected within the shooting range of the wide area camera 110, the PTZ camera 120 starts a tracking process for the moving object 405. Thereafter, the moving object 405 stops at a parking spot, and the recognition process is completed at that spot by the PTZ camera 120 (for example, recognition of a license plate is completed). That is, FIG. 5(a) shows a case where the recognition process is completed within the shooting range of the wide area camera 110.

[0049] During tracking processing, a detected moving object 405 is displayed surrounded by a dashed frame 501 indicating a detection frame. Dashed line 502 indicates the route traveled by moving object 405. A solid line recognition completion frame 503 is superimposed on the display area 402 as an area display notifying the position where recognition has been completed. Note that dashed frame 501 and dashed line 502 are optional and do not need to be displayed. Also, a dot indicating the position where recognition has been completed may be displayed as an area display notifying the position where recognition has been completed.

[0050] 5B is a diagram showing a state when the recognition process of the PTZ camera 120 is completed outside the shooting range of the wide area camera 110. Specifically, after a moving object 405 is detected within the shooting range of the wide area camera 110, the PTZ camera 120 starts a tracking process for the moving object 405. After that, the moving object 405 moves out of the shooting range of the wide area camera 110 from the upper right of the shooting range, moves outside the shooting range (indicated by a broken line 502), and stops at a parking spot. Then, a recognition completion frame 503 shows a state where the recognition process is completed by the PTZ camera 120 at the parking spot (for example, recognition of a license plate is completed). That is, FIG. 5B shows a case where the recognition process is completed outside the shooting range of the wide area camera 110.

[0051] In this manner, in this embodiment, when tracking processing and recognition processing are performed by the PTZ camera 120 outside the shooting range of the wide area camera 110, the position where the tracking processing and recognition processing were performed is displayed in association with the shooting range of the wide area camera 110.

[0052] 6 is a diagram for explaining the display form of the recognition processing result. Fig. 6(a) shows the display result in the area 406 in Fig. 4(a), and Fig. 6(b) shows the display result in the area 407 in Fig. 4(b). The area 601 represented by diagonal lines is a given figure (here, a diagonal line rectangle) showing the result of the recognition processing. In other words, the area 601 is a partial area of ​​the shooting area of ​​the PTZ camera 120.

[0053] The location where the recognition process is completed is acquired as coordinate (X, Y) information from the measurement value in the determination unit 210. Here, the coordinate (X, Y) is a coordinate based on the shooting range (area 402) of the wide area camera 110 (X is the pan direction, and Y is the tilt direction). In FIG. 6(a), the coordinate (X, Y) is expressed simply to show the direction where the recognition is completed. For example, if the size of the frame of the area 601 is constant and detection is made in the up and down directions, it is expressed by corresponding only to the X axis side of the coordinates. For the left and right directions, only the Y axis side is corresponding.

[0054] The center of the region 402 is set to coordinates (0,0), and each vertex of the region 402 is set to the upper left end (-10,5), the upper right end (10,5), the lower left end (-10,-5), and the lower right end (10,-5). In FIG. 6(a), if the region for which recognition processing has been completed has coordinates (X,Y)=(1,-10), the region 601 is displayed at the position (1,-5) (i.e., a position adjacent to the region 402). On the other hand, in FIG. 6(b), the coordinates (X,Y) are more accurately expressed, and if the region for which recognition processing has been completed has coordinates (X,Y)=(1,-10), the region 601 is displayed at the position (1,-10) (i.e., a position away from the region 402).

[0055] <Device Operation> 7 is a flowchart explaining the operation of the imaging system. In the following, it is assumed that the above-mentioned frame generation function, subject identification function, metadata creation function, etc. are performed by each functional unit in the control unit 130 of the imaging device 100, and the processing results are acquired by the terminal 200. However, it may be configured so that some or all of these processes are performed by the CPU 201 of the terminal 200. It is assumed that the terminal 200 acquires images (here, moving images at 15 frames / second) captured by the wide area camera 110 and the PTZ camera 120 in real time via the IP network 300.

[0056] In S701, the control unit 130 detects a moving object 405 from an image captured by the wide area camera 110. Specifically, the subject detection unit 133 detects an image area of ​​the moving object included in the image captured by the wide area camera 110. Any known method can be used for the detection process.

[0057] In S702, the control unit 130 performs tracking and photographing of the moving object 405 detected in S701 with the PTZ camera 120. That is, the drive control unit 131 performs pan-tilt driving so that the photographing direction of the PTZ camera 120 is the direction of the moving object 405. The control unit 130 performs recognition processing (such as recognition of the license plate) of the moving object 405 while performing tracking and photographing of the moving object 405 with the PTZ camera 120. Note that if the moving object being tracked and photographed moves out of the photographable range of the PTZ camera 120, the tracking and photographing of the moving object is terminated and the next moving object is detected.

[0058] In S703, the terminal 200 determines whether or not the recognition process by the PTZ camera 120 is complete. Specifically, the terminal 200 determines whether or not information indicating that the recognition process by the PTZ camera 120 is complete is received from the imaging device 100. If the recognition process is not complete, the terminal 200 continues tracking shooting with the PTZ camera 120. On the other hand, if the recognition process is complete, the process proceeds to S704.

[0059] In S704, the terminal 200 judges whether or not the position where the recognition process of the moving object 405 by the PTZ camera 120 is completed is outside the shooting range of the wide area camera 110. For example, the judgment unit 210 judges whether or not the current shooting direction (optical axis direction) of the PTZ camera 120 is outside the shooting range of the wide area camera 110 from the pan and tilt drive information of the PTZ camera 120 received from the imaging device 100. Alternatively, the judgment is made based on the subject information and the result information of the recognition process by the camera judgment unit 132 received from the imaging device 100. For example, if the moving object 405 is not detected by the wide area camera 110 at the time when the recognition process by the PTZ camera 120 is completed, it is judged that the moving object 405 is outside the shooting range of the wide area camera 110. If the moving object 405 is outside the shooting range of the wide area camera 110, the process proceeds to S705, and if the moving object 405 is within the shooting range of the wide area camera 110, the process proceeds to S708.

[0060] In S705, the terminal 200 acquires position information of the subject in the captured image obtained by the PTZ camera 120. If the geographical position is known by a GPS or the like, the position information of the subject may be acquired by adding zoom information to the pan / tilt drive information.

[0061] In S706, the terminal 200 calculates the coordinates (X, Y) of the subject based on the shooting range of the wide area camera 110 from the position information acquired in S705.

[0062] In S707, the terminal 200 displays the recognition completion frame 503 of the subject inside the area 406 (or area 407) and outside the display area 402 based on the result of S706. That is, the position of the subject is displayed based on the shooting range (area 402) of the wide-area camera 110 (FIGS. 5(b), 6(a), and 6(b)). The position of the subject (inside the recognition completion frame 503) may be simplified and displayed as a rectangle like the area 601, or the captured image (or thumbnail image) at the time of recognition completion may be displayed. Also, when the mouse cursor is placed over the position of the subject, the image may be displayed in pop-up format for a certain period of time. Note that the captured image to be displayed is the captured image at the time of recognition completion, so that only the image to be recognized may be displayed. For example, when a license plate recognition process is performed, a "license plate" image is displayed, when a person recognition process is performed, a "whole person's body" image is displayed, and when a face recognition process is performed, a "face" image is displayed. Also, the image to be displayed may be changed according to the type of recognition process.

[0063] In S708, the terminal 200 displays the recognition-completed frame 503 of the subject inside the display area 402 (FIG. 5(a)). Note that in S708, since the recognition-completed frame 503 is displayed in the same place as the dashed frame 501, it is desirable to display the recognition-completed frame 503 and the dashed frame 501 in different forms (shapes, colors).

[0064] In S709, as a process after displaying the recognition completion frame 503 in S707 or S708, the terminal 200 ends reception of pan and tilt drive information from the PTZ camera 120. For example, reception is stopped when the wide area camera 110 or the PTZ camera 120 detects that the subject has stopped moving (the car has stopped).

[0065] As described above, according to the first embodiment, in a system combining a wide-area camera and a PTZ camera, images captured by the respective cameras are displayed so that the relationship between the images can be understood. In particular, when the PTZ camera 120 captures an image outside the capture range of the wide-area camera 110, a display showing the capture range of the PTZ camera 120 is performed outside the capture range of the wide-area camera 110. This allows the user to easily understand the relationship between the captured images (the positional relationship of the capture ranges).

[0066] Second embodiment In the second embodiment, another display form showing the relationship between the images captured by the wide-area camera and the PTZ camera in a system combining the wide-area camera and the PTZ camera will be described. In the first embodiment, the captured image 803 at the time of completion of the recognition process by the PTZ camera 120 is displayed. In that case, the image captured by the wide-area camera 110 and the image captured by the PTZ camera 120 may look significantly different. For example, when the wide-area camera 110 displays the entire vehicle as a subject and the PTZ camera 120 displays only the license plate as a subject, the user cannot easily recognize whether these images are correctly linked. Therefore, in the second embodiment, a form that allows the user to more easily recognize that the image captured by the PTZ camera 120 at the time of completion of the recognition process is the subject to be tracked will be described. Specifically, an example will be described in which an image when the subject moves from within the capture range of the wide-area camera 110 to outside the capture range can be further displayed.

[0067] <Control of the screen displayed on the display device 208 of the terminal 200> Fig. 8 is a diagram showing an example of a screen showing the recognition processing result in the second embodiment. As with Fig. 5(b) in the first embodiment, this is a diagram showing a state when the recognition processing of the PTZ camera 120 is completed outside the shooting range of the wide area camera 110. The difference from Fig. 5(b) is that when the moving object 405 passes through the upper hatched area 802, an image 801 of the moving object 405 captured by the wide area camera 110 is acquired and displayed. Fig. 8 shows an example of passing through the upper hatched area 802, but the same processing is performed when passing through the lower hatched area 802.

[0068] The captured image 801 is stored in association with a captured image 803 captured upon completion of the subsequent recognition process. For example, the captured images (captured images 801, 803) can be linked by describing identification information for identifying each captured image as metadata for each captured image. However, any known linking method can be used. Note that the captured image 803 can be an image captured upon completion of recognition of the moving object 405 by the PTZ camera 120, or an image area of ​​the recognition target (such as a license plate).

[0069] The two hatched areas 802 shown in FIG. 8 are areas that are set to detect a moving object going out of the shooting range of the wide area camera 110. That is, the hatched area 802 is set to obtain the latest image of the moving object captured by the wide area camera 110. For example, before operating the imaging device 100 in the monitoring mode, the user refers to the captured image of the wide area camera 110 displayed on the display device 208 and sets the hatched area 802 via the operation unit 206. When the imaging device 100 starts operating in the monitoring mode, the terminal 200 monitors the video from the wide area camera 110 and stores the captured image (detected image) at the time when the moving object passes through the hatched area 802 in the HDD 204. The stored captured image is to be used as the captured image 801. Note that the hatched area 802 only needs to be displayed on a setting screen for setting the area, and does not need to be displayed on the display image of the wide area camera 110 displayed on the display device 208 while the imaging device 100 is operating in the monitoring mode, for example.

[0070] As described above, according to the second embodiment, the captured image immediately before the moving object to be tracked goes out of the capture range of the wide area camera 110 is stored in association with the captured image 803 at the time of completion of the recognition process by the PTZ camera 120. This allows the user to check the captured image by the wide area camera 110, which allows the user to grasp the relationship with surrounding subjects, scenery, etc., and therefore makes it easier to recognize that the captured image 803 is the subject to be tracked.

[0071] In the above description, the captured image when the moving object goes out of the imaging range of the wide area camera 110 is acquired by setting the hatched area 802, but other acquisition methods may be used. For example, the captured image may be acquired when the moving object to be tracked reaches the outer periphery of the imaging range of the wide area camera 110.

[0072] The disclosure of this specification includes the following display control device, display control method, program, and imaging system. (Item 1) A display control device that controls display of a captured image acquired by an imaging device having a first imaging unit that captures a predetermined area and a second imaging unit that can capture an area range that includes at least the predetermined area by changing and controlling the imaging direction, an image acquisition means for acquiring a first captured image acquired by the first imaging section and a second captured image acquired by the second imaging section; a display control means for displaying, on a display unit, the first captured image and an area display corresponding to the capture area of ​​the second imaging unit; having When the second captured image is an image captured outside the predetermined area, the display control means displays the area display corresponding to the capture area of ​​the second imaging unit in an area outside the area where the first captured image is displayed on the display unit. Display control device. (Item 2) When the second captured image is an image captured inside the predetermined area, the display control means displays the area display corresponding to the capture area of ​​the second imaging unit in a superimposed manner within an area displaying the first captured image on the display unit. 2. The display control device according to item 1, (Item 3) The method further includes an information acquisition unit for acquiring information regarding a shooting direction of the second imaging unit when the second photographed image is photographed, The display control means determines whether the second captured image is an image captured outside the predetermined area or an image captured inside the predetermined area based on the information. 3. The display control device according to item 1 or 2. (Item 4) The display control means displays a predetermined figure as the area display. 4. The display control device according to any one of items 1 to 3. (Item 5) The second imaging unit is configured to track and capture a moving object detected in the first captured image obtained by the first imaging unit, The second imaging unit is configured to perform a predetermined recognition process on the moving object, The display control means displays, as the area display, a recognition target image when the second imaging unit completes the predetermined recognition process for the moving object. 5. A display control device according to any one of items 1 to 4. (Item 6) The moving object is a vehicle, and the predetermined recognition process is a recognition process of a license plate of the vehicle, or The moving object is a person, and the predetermined recognition process is a face recognition process. 6. A display control device according to item 5, characterized in that: (Item 7) A designation means for designating a detection area within the predetermined area for detecting passage of a moving object; a storage means for storing a detection image captured when the moving object passes through the detection area while the second imaging unit is performing a tracking process of the moving object, in association with the second captured image; and The display control means displays the detected image on the display unit based on an instruction from a user. 7. The display control device according to item 5 or 6, (Item 8) The area display is a display corresponding to a partial area of ​​the photographing area of ​​the second imaging unit. 8. The display control device according to any one of items 1 to 7, (Item 9) A display control method for controlling display of a captured image acquired by an imaging device having a first imaging unit that captures a predetermined area and a second imaging unit that can capture an area range that includes at least the predetermined area by changing and controlling an imaging direction, comprising: an image acquiring step of acquiring a first captured image acquired by the first imaging unit and a second captured image acquired by the second imaging unit; a display control step of displaying, on a display unit, the first captured image and an area display corresponding to the capture area of ​​the second imaging unit; Including, In the display control step, when the second captured image is an image captured outside the predetermined area, the area display corresponding to the capturing area of ​​the second imaging unit is displayed in an area outside the area where the first captured image is displayed on the display unit. Display control method. (Item 10) A program for causing a computer to execute the display control method according to item 9. (Item 11) a first imaging means for imaging a predetermined area; a second imaging means capable of imaging an area range including at least the predetermined area by changing and controlling an imaging direction; an image acquisition means for acquiring a first photographed image acquired by the first imaging means and a second photographed image acquired by the second imaging means; a display control means for displaying, on a display unit, the first captured image and an area display corresponding to the capture area of ​​the second imaging means; having When the second captured image is an image captured outside the predetermined area, the display control means displays the area display corresponding to the capture area of ​​the second imaging means in an area outside the area where the first captured image is displayed on the display unit. Imaging system.

[0073] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0074] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0075] 100 imaging device; 200 terminal; 300 IP network; 110 wide-area camera; 120 PTZ camera

Claims

1. A display control device that controls display of a captured image acquired by an imaging device having a first imaging unit that captures a predetermined area and a second imaging unit that can capture an area range that includes at least the predetermined area by changing and controlling the imaging direction, an image acquisition means for acquiring a first captured image acquired by the first imaging section and a second captured image acquired by the second imaging section; a display control means for displaying, on a display unit, the first captured image and an area display corresponding to a capture area of ​​the second image capture unit; having When the second captured image is an image captured outside the predetermined area, the display control means displays the area display corresponding to the capture area of ​​the second imaging unit in an area outside the area where the first captured image is displayed on the display unit. Display control device.

2. When the second captured image is an image captured inside the predetermined area, the display control means displays the area display corresponding to the capture area of ​​the second imaging unit in a superimposed manner within an area displaying the first captured image on the display unit. The display control device according to claim 1 .

3. The camera further includes an information acquisition unit that acquires information about a shooting direction of the second imaging unit when the second photographed image is photographed, The display control means determines whether the second captured image is an image captured outside the predetermined area or an image captured inside the predetermined area based on the information. The display control device according to claim 1 .

4. The display control means displays a predetermined figure as the area display. The display control device according to claim 1 .

5. The second imaging unit is configured to track and capture a moving object detected in the first captured image obtained by the first imaging unit, The second imaging unit is configured to perform a predetermined recognition process on the moving object, The display control means displays, as the area display, a recognition target image when the second imaging unit completes the predetermined recognition process for the moving object. The display control device according to claim 1 .

6. The moving object is a vehicle, and the predetermined recognition process is a recognition process of a license plate of the vehicle, or The moving object is a person, and the predetermined recognition process is a face recognition process.

6. The display control device according to claim 5.

7. A designation means for designating a detection area within the predetermined area for detecting passage of a moving object; a storage means for storing a detection image captured when the moving object passes through the detection area while the second imaging unit is performing a tracking process of the moving object, in association with the second captured image; and The display control means displays the detected image on the display unit based on an instruction from a user.

6. The display control device according to claim 5.

8. The area display is a display corresponding to a partial area of ​​the photographing area of ​​the second imaging unit. The display control device according to claim 1 .

9. A display control method for controlling display of a captured image acquired by an imaging device having a first imaging unit that captures a predetermined area and a second imaging unit that can capture an area range including at least the predetermined area by changing and controlling a capturing direction, comprising: an image acquiring step of acquiring a first captured image acquired by the first imaging unit and a second captured image acquired by the second imaging unit; a display control step of displaying the first captured image and an area display corresponding to the capture area of ​​the second image capture unit on a display unit; Including, In the display control step, when the second captured image is an image captured outside the predetermined area, the area display corresponding to the capture area of ​​the second imaging unit is displayed in an area outside the area where the first captured image is displayed on the display unit. Display control method.

10. A program for causing a computer to execute the display control method according to claim 9.

11. a first imaging means for imaging a predetermined area; a second imaging means capable of imaging an area including at least the predetermined area by changing and controlling an imaging direction; an image acquisition means for acquiring a first photographed image acquired by the first imaging means and a second photographed image acquired by the second imaging means; a display control means for displaying, on a display unit, the first captured image and an area display corresponding to an image capturing area of ​​the second image capturing means; having When the second captured image is an image captured outside the predetermined area, the display control means displays the area display corresponding to the capture area of ​​the second imaging means in an area outside the area where the first captured image is displayed on the display unit. Imaging system.

Images