Imaging apparatus, method for controlling imaging apparatus, program, and imaging system

Abstract

To provide an imaging apparatus that identifies a person who designates a user considering a delay on a network.SOLUTION: An imaging apparatus comprises imaging means that picks up video images, and the imaging apparatus has: transmission means that transmits the picked-up video images to a client device; receiving means that receives a signal for setting a designation area in an object to be tracked from the client device; acquisition means that, on the basis of a delay time in communication between the imaging apparatus and the client device, acquires a past frame image from a plurality of frame images included in the video images; identification means that identifies the object to be tracked from the designation area in the past frame image; and angle-of-view changing means that, when the object to be tracked identified by the identification means is present in the latest frame image of the plurality of frame images included in the video images, changes an angle of field of the imaging apparatus to track the object to be tracked.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a control method for an image capture device, a program, and an image capture system. [Background technology]

[0002] Currently, with the advancement of network camera (hereafter referred to as camera) technology, products are being sold that have the function of detecting people inside the camera and tracking the detected people using a pan-tilt (hereafter referred to as PT) mechanism. With such cameras, the user can specify a person in the image, and the camera will continue to detect the target person in each image frame and control the PT so that the target person is in the center of the image.

[0003] For example, in Patent Document 1, the aim is to improve the accuracy of tracking a person by predicting the moving direction of the target person from the past trajectory of the person and performing PT control to face in that direction. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-198470 A Summary of the Invention [Problem to be solved by the invention]

[0005] When specifying a person from within a video, network delays occur in both the request from the client to the camera and the response from the camera to the client. This network delay can cause the person in the area specified by the user on the client to differ from the person in the captured image when the specified area is received by the camera. This can lead to tracking a person different from the person specified by the user.

[0006] Therefore, an object of the present invention is to provide an imaging device that identifies a person designated by a user, taking into consideration delays on a network. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, an imaging device as one aspect of the present invention is an imaging device equipped with an imaging means for capturing video, and is characterized in having: a transmitting means for transmitting the captured video to a client device; a receiving means for receiving a signal from the client device for setting a designated area in the tracking target; an acquiring means for acquiring a past frame image from a plurality of frame images included in the video based on a delay time in communication between the imaging device and the client device; an identifying means for identifying the tracking target from the designated area in the past frame image; and an angle of view changing means for changing the angle of view of the imaging device in order to track the tracking target if the tracking target identified by the identifying means is present in the latest frame image among the plurality of frame images included in the video. Effect of the Invention

[0008] According to the present invention, it is possible to provide an imaging device that identifies a person designated by a user, taking into consideration delays on a network. [Brief description of the drawings]

[0009] [Figure 1] 1 is a configuration diagram illustrating an example of an entire system according to a first embodiment. [Diagram 2] 1 is a configuration diagram illustrating an example of an imaging device according to a first embodiment. [Diagram 3] FIG. 2 is a configuration diagram illustrating an example of a control device according to the first embodiment. [Figure 4] FIG. 4 is a block diagram showing an example of a program of the imaging device in the first embodiment. [Diagram 5] FIG. 4 is a block diagram illustrating an example of a program of a client device according to the first embodiment. [Figure 6] FIG. 2 is a schematic diagram illustrating an example of a process according to the first embodiment. [Figure 7]4 is a flowchart illustrating an example of a process according to the first embodiment. [Figure 8] FIG. 11 is a block diagram showing an example of a program of the imaging device according to the second embodiment. [Figure 9] 13 is a flowchart illustrating an example of a process according to a second embodiment. [Figure 10] FIG. 11 is a schematic diagram illustrating an example of a process according to the second embodiment. [Figure 11] FIG. 11 is a configuration diagram showing an example of an entire system according to a third embodiment. [Figure 12] FIG. 11 is a configuration diagram illustrating an example of an imaging device according to a third embodiment. [Figure 13] FIG. 11 is a configuration diagram illustrating an example of an MEC server in a third embodiment. [Figure 14] FIG. 11 is a block diagram showing an example of a program of the imaging device according to the third embodiment. [Figure 15] FIG. 11 is a configuration diagram showing an example of a program of an MEC server in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the following embodiments. The same or equivalent components, parts, and processes shown in each drawing are given the same reference numerals, and duplicated explanations are omitted as appropriate. In addition, some of the parts that are not important for the explanation are omitted in each drawing.

[0011] <Example 1> 1 is a diagram showing a system configuration according to Example 1. The system in Example 1 includes a network 101, a camera 102, and a client device 103.

[0012] The network 101 is a network that connects the camera 102 and the client device 103. The network 101 is realized by a plurality of routers, switches, cables, etc. that comply with a communication standard such as ETHERNET (registered trademark). Note that the network 101 may also be realized by the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless Lan), a WAN (Wide Area Network), etc.

[0013] The camera (imaging device) 102 captures image data and video data through an imaging lens. In the first embodiment, the camera 102 transmits captured image data and the like in response to a request from a client via the network 101. However, this is not limiting, and the camera 102 may actively transmit image data to an external device such as a client device 103 that is connected in advance. The client device (information processing device, control device) 103 is an information terminal such as a personal computer (PC), a tablet, or a smartphone.

[0014] 2 is a diagram showing an example of a schematic configuration of the camera 102 in the first embodiment. The camera 102 has an internal bus 201 configured therein. The camera 102 is configured to have a CPU 202, a primary storage device 203, a secondary storage device 204, an image input interface (I / F) 205, an imaging unit 206, a PTZ driving unit 207, an image recognition unit 208, and a network I / F 209. The CPU 202, the primary storage device 203, the secondary storage device 204, the image input interface (I / F) 205, the imaging unit 206, the PTZ driving unit 207, the image recognition unit 208, and the network I / F 209 are connected to the internal bus 201.

[0015] A CPU (Central Processing Unit) 202 is a central processing unit that performs overall control of the operation of the camera 102 by executing a control program stored in a primary storage device 203 .

[0016] The primary storage device 203 is, for example, a writable high-speed storage device such as a RAM. For example, the OS (operating system), various control programs, and various data are loaded into the primary storage device 203, and the primary storage device 203 is also used as a working area when the CPU 202 executes the OS and various programs.

[0017] The secondary storage device 204 is a non-volatile storage device such as a HDD, a flash memory, or an SD card. The secondary storage device 204 may be configured to be removable. The secondary storage device 204 is used as a permanent storage area for an OS, various programs, various data, etc., and is also used as a storage area for various short-term data, etc. In the first embodiment, the secondary storage device 204 also functions as a storage unit, and stores image data, video data including a plurality of frames, and information on a delay time, which will be described later.

[0018] The image input I / F 205 is an interface for inputting image data from an imaging unit 206 connected by wire or wirelessly.

[0019] The imaging unit 206 has an imaging lens, an imaging sensor such as a CCD or a CMOS, and captures an image via the imaging lens.

[0020] The PTZ driving unit 207 controls a motor or the like to pan, tilt, or zoom the camera 102. That is, the PTZ driving unit 207 is capable of controlling the PTZ (pan, tilt, zoom) of the camera 102. Note that the PTZ driving unit 207 is not limited to only controlling the PTZ, and may also set and control the camera 102, such as focus and white balance. The PTZ driving unit 207 may also be a camera control unit that manages these controls.

[0021] The image recognition unit 208 performs image recognition on the image captured by the imaging unit 206, and issues instructions (sends a specified control signal) to the PTZ driving unit 207 to cause the PTZ driving unit 207 to perform PTZ control of the camera 102 in order to fit the specified target within the angle of view.

[0022] The network I / F 209 is an interface for connecting to the network 101. In the first embodiment, network communication from the client device 103 is transmitted to various programs via the network I / F 209.

[0023] 3 is a diagram showing an example of a schematic configuration of the client device 103 in the embodiment 1. The client device 103 has an internal bus 301 therein. The client device 103 is configured to have a CPU 302, a primary storage device 303, a secondary storage device 304, a user input / output I / F 305, and a network I / F 306, which are connected to the internal bus 301.

[0024] The CPU 302 is a central processing unit, and performs overall control of the operations in the client device 103 by executing a control program stored in the primary storage device 303 .

[0025] The primary storage device 303 is, for example, a writable high-speed storage device such as a RAM. For example, the OS (operating system), various programs, and various data are loaded into the primary storage device 303, and the primary storage device 303 is also used as a working area when the CPU 302 executes the OS and various programs.

[0026] The secondary storage device 304 is a non-volatile storage device such as a HDD, a flash memory, or an SD card. The secondary storage device 204 may be configured to be removable. The secondary storage device 304 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0027] The user input / output I / F 305 is an I / F for the client device 103 to receive operations from the user, input user information, and output images and the like to the user. Examples of the user input / output I / F 305 include display devices (display means) such as a display and a touch panel, and operation means such as a keyboard, a mouse, and a microphone. Here, the touch panel and the display may be configured as an integrated type such as a touch panel-mounted display that can detect the touch position. These I / Fs are used for touch and mouse operations, voice input, processing using captured images, and the like.

[0028] The network I / F 306 is an interface for connecting to the network 101. Network communication from the camera 102 is transmitted to various programs via the network I / F 306.

[0029] 4 is a configuration diagram showing an example of a program of the camera 102 in the embodiment 1. Specifically, it is a diagram showing an example of various programs and various data stored in the primary storage device 203 and the secondary storage device 204 of the camera 102 in the embodiment 1.

[0030] Into the primary storage device 203, an OS 401, an image capturing program 402, a communication control program 403, a delay time measurement program 404, an image recognition program 405, an object determination program 406, and a PTZ control program 407 are loaded.

[0031] The secondary storage device 204 stores delay time information 408, image buffer data 409, and target image data 410. Also stored are various programs loaded into the primary storage device 203. The various programs referred to here are an OS 401, an image capturing program 402, a communication control program 403, a delay time measurement program 404, an image recognition program 405, a target determination program 406, and a PTZ control program 407. Note that, although the various data (delay time information 408, image buffer data 409, and target image data 410) are stored in the secondary storage device 204 in the first embodiment, in other embodiments, they may be stored in the primary storage device 203 for temporary use.

[0032] The OS 401 is a basic program for controlling the entire camera 102. Here, the positions (addresses) and sizes in the primary storage device 203 of the various programs loaded into the primary storage device 203 are managed by the OS 401.

[0033] In response to an image acquisition request from the client device 103 or an instruction from another program, the image capturing program 402 acquires an image captured by the image capturing unit via the image input I / F 205. In addition, the image capturing program 402 converts the acquired image into image data such as JPEG or video data such as H.264.

[0034] The communication control program 403 is a program for controlling communication with the client device 103 from the network I / F 209 through the network 101. One example of the content of the communication is receiving an acquisition request for an image captured by the imaging unit 206, or receiving a PTZ control request for the PTZ driving unit 207 of the camera 102. Note that the request received from the client device 103 may use a general communication protocol such as http (hypertext transfer protocol).

[0035] The delay time measurement program 404 is a program for measuring a delay time in communication with the client device 103. One example of a method for measuring the delay time is to use an average of round trip times in network communication between the camera 102 and the client device 103 as the delay time. However, the method is not limited to this, and other measurement methods may be used as long as they can measure the delay time with the client device 103.

[0036] The image recognition program 405 is a program that performs image recognition on image data such as captured images and image buffer data 409, which will be described later. Specifically, the image recognition program 405 recognizes an object determined by an object determination program 406, which will be described later, from within the image data. Thereafter, based on the recognition result as necessary, the image recognition program 405 instructs a PTZ control program 407, which will be described later, to perform PTZ control. In the first embodiment, the description will be based on a function for tracking an object, but the present invention is not limited to this, and various other analysis functions may be provided.

[0037] The target determination program 406 is a program that determines a tracking target from image buffer data 409, which will be described later, based on the delay time measured by the delay time measurement program 404. For example, when there is a round-trip delay time of 0.1 seconds between the camera 102 and the client device 103, the target determination program 406 reads an image 0.1 seconds before the latest image from the image buffer data 409, and determines a target from among the image. Note that, although the tracking target is assumed to be a person in the first embodiment, it is not limited to this and may be an animal, a robot, or the like.

[0038] The PTZ control program 407 performs PTZ control on the PTZ driving unit 207 based on various programs loaded into the primary storage device 203 and PTZ control instructions from the client device 103 .

[0039] The delay time information 408 is information on the delay time between the camera 102 and the client device 103 measured by the delay time measurement program 404. Here, when a plurality of client devices 103 are connected to the camera 102, a unique ID may be assigned to each client device 103, and the delay time with each client device 103 may be stored.

[0040] The image buffer data 409 is image data captured by the image capturing program 402 for a time measured by the delay time measuring program 404. As described above, when multiple client devices 103 are connected, the image buffer data 409 is stored according to the client device 103 having the longest delay time among them. The target image data 410 is image data (frame image) of the tracking target determined by the target determination program 406.

[0041] FIG. 5 is a diagram showing an example of various programs and various data stored in the primary storage device 203 and the secondary storage device 204 of the client device 103 in the first embodiment.

[0042] An OS 501 , a communication control program 502 , a display program 503 , and an area designation program 504 are loaded into the primary storage device 303 .

[0043] Region information 505 is stored in the secondary storage device 304. In this embodiment, the region information 505 is stored in the secondary storage device 304, but in other embodiments, it may be stored in the primary storage device 303 as primary information.

[0044] The OS 501 is a basic program for controlling the entire client device 103. Here, the positions (addresses) and sizes in the primary storage device 203 of various programs (the OS 501, the communication control program 502, the display program 503, and the area designation program 504) loaded into the primary storage device 203 are managed by the OS 501.

[0045] The communication control program 502 is a program for controlling communication with the camera 102 from the network I / F 306 through the network 101. One example of the content of the communication is sending an acquisition request for an image captured by the imaging unit 206 of the camera 102, or sending a PTZ control request to the PTZ driving unit 207 of the camera 102. Note that the request sent to the camera 102 may use a general communication protocol such as http (hypertext transfer protocol).

[0046] The display program 503 is a program that displays image data and video data acquired from the camera 102, icons for controlling the camera 102, and the like on a display device that is one of the user input / output I / Fs 305 of the client device 103. In addition, the display program 503 can receive input from an external device such as a mouse or keyboard, or, if the display device is a touch display, touch operations and the like as input from the user.

[0047] The area designation program 504 is a program that designates an area for image data captured by the camera 102, based on a user's input input to the user input / output I / F 305. Here, the user can designate an area by designating surrounding designated pixels of selected coordinates as an area, or by designating multiple (three or more) coordinates and designating the area inside the selected coordinates, and either method may be used in the first embodiment.

[0048] Area information 505 is information indicating an area (designated area) designated by the user using the user input / output I / F 305, etc. For example, if the designated area is rectangular, it may be the coordinates of four points, or a reference point and vertical and horizontal widths, but the storage format is not important. Note that the designated area is not limited to a rectangle, and the user can designate an area that is circular, elliptical, or polygonal using the user input / output I / F 305, etc.

[0049] 6 is a diagram showing the relationship between an image captured by the camera 102 at a certain time t5 and an image displayed on the client device 103 at the same time. In this example, the delay on the network from the camera 102 to the client device 103 and from the client device 103 to the camera 102 is t n -t n-2 This indicates that there are seconds remaining.

[0050] A camera captured image 601 indicates an image captured by the camera 102 at each time t. Taking Fig. 6 as an example, it can be seen that only one person is captured at time t1 and time t2, but a second person is captured in addition to the first person from time t3 onwards (t3, t4, t5).

[0051] The client display image 602 indicates an image displayed on the client device 103 at each time t. For example, in FIG. 6, when an image is transmitted from the camera 102 to the client device 103, n -t n-2 Because there is a delay of 1 second, at the same time t n -t n-2 The images are displayed on the client device 103 with a delay of one second.

[0052] The user-specified area 603 is an area (specified area) specified by the user as a tracking target. When the client device 103 transmits information about the user-specified area 603 to the camera 102, the camera 102 sets (determines) the target shown in the user-specified area 603 as the tracking target.

[0053] The camera-specified area 604 is the area at the timing when the camera 102 receives the user-specified area 603 from the client device 103. In the example of FIG. 6, the client device 103 sets the user-specified area 603 at time t3, and then transmits information about the user-specified area 603 to the camera 102. Here, after the camera receives the information about the user-specified area, a predetermined time (t n -t n-2Since a delay of 10 seconds occurs, another person is present in the specified area on the image captured by camera 102. That is, a person who appears in user-specified area 603 set at time t3 is desired to be the tracking target, but when that information is transmitted to camera 102, a person who is not the person the user wants to be the tracking target is present in user-specified area 603 at time t5. Therefore, that person is tracked as the tracking target. As a result, a person who is not the person the user wants to be the tracking target becomes the tracking target, and an unintended person may become the tracking target.

[0054] The tracking target area 605 is a tracking target area set (determined) by the target determination program 406 based on the delay time between the camera 102 and the client device 103 measured by the delay time measurement program 404 and the user-specified area 603. In the first embodiment, the delay time between the camera 102 and the client device 103 is t n -t n-4 This means that the time from when the camera 102 receives the area specification from the client device 103 is t n -t n-4 In the frame (frame image) seconds before, a tracking target area 605 is set and the target to be tracked (tracking target) is specified. After the target to be tracked is specified, the target reflected in the tracking target area 605 is determined as the tracking target and is stored as target image data 410 in the secondary storage device 203 of the camera 102.

[0055] Hereinafter, a flow from when the camera 102 receives a designation of an area from the client device 103 to when the camera 102 tracks a tracking target will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of processing in the first embodiment. Each of the following processes is realized by the CPU 202 of the camera 102 executing a program stored in the primary storage device 203. However, this is merely an example, and some or all of the processes described below may be realized not only by the camera 102 but also by the CPU 302 of the client device 103 or dedicated hardware. Note that before the start of step S701, the camera 102 transmits a video that has been captured or is being captured to the client device 103.

[0056] First, in step S701, the object determination program 406 receives area designation operation information by the user to the user input / output I / F 305 of the client device 103 via the network 101. That is, the object determination program 406 receives information on the user designated area 603 that has been set by the user based on the image transmitted from the camera 102.

[0057] Next, in step S702, the object determination program 406 acquires and reads the delay time information 408 stored in the secondary storage device 204 of the camera 102. In this process, the delay time information 408 is the total time between the time until the video arrives at the client device 103 and the time until the camera 102 receives a predetermined signal that designates the user-designated area 603 using the video that has arrived.

[0058] Next, in step S703, the object determination program 406 reads data of a frame (past frame) that is the delay time before the current time from the image buffer data 409 based on the delay time information 408. During this process, the object determination program 406 also functions as an acquisition unit that acquires a frame that is the delay time before the current time from among multiple frames included in the video based on the delay time information 408.

[0059] Next, in step S704, the image recognition program 405 specifies the tracking target based on the frame data read in step S703 and the operation information (information of the user specified area 603 specified by the user) received in step S701. Specifically, based on the information of the user specified area 603 specified by the user in step S702 and the delay time information 408, the tracking target area (specified area) 605 is set in the frame preceding the delay time, and the tracking target is specified from this tracking target area 605. In this process, the image recognition program 405 functions as a specifying means for specifying the tracking target from the tracking target area 605 in the frame preceding the delay time.

[0060] Here, as a method for identifying the tracking target, a method for identifying a moving object based on frame difference information within the region may be used. Alternatively, a method for identifying the tracking target may be used using a method such as semantic region division, which divides a specified region according to the meaning in the image data using a method such as Deep Learning. Alternatively, if the tracking target is a specific object such as a person, person detection by pattern matching may be used, but any method may be used, or other methods may be used.

[0061] Next, in step S705, the image recognition program 405 determines by image recognition whether the tracking target identified in S704 exists in the latest (current) frame among the multiple frames included in the video. If the result of the determination is that the tracking target identified in S704 exists in the latest frame, it is determined that the tracking target identified in S704 has been detected, and the process proceeds to step S706. On the other hand, if the tracking target identified in S704 does not exist in the latest frame, it is determined that the tracking target identified in S704 has not been detected, and the process proceeds to step S707. Note that the determination method may be any detection method, including pattern matching, and the method is not limited.

[0062] Next, in step S706, in order to track the detected tracking target, the image recognition program 405 issues a PTZ control instruction to the PTZ control program 407 so that the tracking target does not deviate from the angle of view of the camera 102. In this process, the image recognition program 405 functions as an angle of view changing unit that changes the angle of view of the camera 102.

[0063] Next, in step S707, the image recognition program 405 calculates (or detects) the trajectory (movement trajectory) of the tracking target from the frame in which the tracking target was determined to the latest frame. The calculation of the movement trajectory involves, for example, checking whether the tracking target exists in each frame, and recording its coordinates if it exists, thereby making it possible to know how the tracking target moved. In this process, the image recognition program 405 functions as a first calculation means for calculating the trajectory of the tracking target from the past frame in which the target was determined to the latest frame.

[0064] Next, in step S708, the image recognition program 405 estimates (predicts) the direction in which the tracking target is currently located based on the trajectory information calculated in step S707, and calculates the angle of view in which the estimated person moves, i.e., the estimated moving direction of the person. For example, based on the coordinate information recorded in step S707, it is possible to calculate the moving direction by using a Bezier curve or the like so that all coordinates are connected. In the calculation, it is possible to improve the accuracy by using accompanying information such as the angle of view information and installation position information of the camera 102. In this process, the image recognition program 405 functions as a second calculation means that estimates the current position of the tracking target based on the trajectory information of the tracking target, and calculates the moving direction of the tracking target based on the estimated position of the tracking target.

[0065] Next, in step S709, the image recognition program 405 transmits a PTZ control instruction (a predetermined control signal) to the PTZ control program 407 in order to control the PTZ position calculated in step S708. The PTZ control program 407 that has received the PTZ control instruction controls the PTZ driving unit 207 in the instructed direction. Next, in step S710, the image recognition program 405 detects the tracking target from the latest frame.

[0066] Next, in step S711, the image recognition program 405 determines whether or not the tracking target has been detected from the latest frame of the multiple frames included in the video. If the result of the determination is that the tracking target has been detected from the latest frame, the program proceeds to step S706. On the other hand, if the tracking target has not been detected from the latest frame, the flow of this process ends. Here, at the end of the flow, the program may return to the position before the movement in step S709. Furthermore, the process at the end depends on the settings of the camera 102.

[0067] As described above, according to the first embodiment, by tracing back the data of a plurality of frames included in the video, it is possible to identify and track the tracking target based on the area specified by the user. As a result, even if something different from the tracking target specified by the user exists in the frame due to a network delay time between the camera 102 and the client device 103, it is possible to identify and track the tracking target specified by the user. Furthermore, by estimating the current position of the tracking target using a plurality of frames, it is possible to track the tracking target even if the tracking target has moved outside the frame (outside the angle of view of the camera 102) in the latest frame. <Example 2> Hereinafter, a system configuration according to the second embodiment will be described with reference to Fig. 8 and Fig. 9. Note that in the second embodiment, the same configurations and descriptions as those of the camera 102 and the client device 103 in the first embodiment will be omitted.

[0068] Fig. 8 is a block diagram showing an example of a program of the camera 102 in the second embodiment. It is a block diagram showing an example of a program configuration of the camera 102 in the second embodiment. The difference detection program 801 shown in Fig. 8 detects a difference between frames as a change between previous and next frames when saving the image buffer data 409. Then, the difference detection program 801 determines whether to save the detected difference as buffer data in the image buffer data 409 depending on the magnitude of the difference.

[0069] 9 is a flowchart showing an example of processing in the second embodiment. Specifically, it is a flowchart showing an example of processing when saving image buffer data 409. Note that each of the following processes is realized by CPU 202 of camera 102 executing a program stored in primary storage device 203. However, this is merely an example, and some or all of the processes described below may be realized not only by camera 102 but also by CPU 302 of client device 103 or dedicated hardware.

[0070] First, in step S901, the difference detection program 801 of the camera 102 acquires a frame (frame image) as captured image data through the image capturing program 402.

[0071] Next, in step S902, the difference detection program 801 compares the frame acquired in S901 (frame image to be compared) with the latest frame stored in the image buffer data 409 to detect the difference between the frames (change between frame images). Then, it is determined whether the detected difference between the frames is large, that is, whether the detected difference exceeds a predetermined threshold. If it is determined that the difference between the frame acquired in S901 and the latest frame stored in the image buffer data 409 is less than the predetermined threshold, the process proceeds to step S903. On the other hand, if it is determined that the difference between the frame acquired in S901 and the latest frame stored in the image buffer data 409 is equal to or greater than the predetermined threshold, the process proceeds to step S904. However, if the image buffer data 409 does not exist, such as immediately after startup, the process proceeds to step S904. In this process, the difference detection program 801 functions as a detection means for detecting the difference between a plurality of frames.

[0072] Here, in step S902, the difference detection program 801 may use a method of comparing pixel values ​​of pixels at the same coordinates as a method of determination. Furthermore, a method of comparing general image features such as HOG (Histograms of Oriented Gradients) features may also be used. Furthermore, any method may be used without being limited to these. The predetermined threshold value may be arbitrarily set by the user, or may be automatically set by the difference detection program 801 according to the frames to be compared, the imaging conditions, etc.

[0073] Next, in step S903, the difference detection program 801 deletes the latest frame (latest image data) of the image buffer data 409 used for comparison, and updates the image buffer data 409. Next, in step S904, the difference detection program 801 saves the frame acquired in step S901 and used for comparison as the image buffer data 409, and updates the image buffer data 409.

[0074] Next, in step S905, the difference detection program 801 determines whether the image buffer data 409 has exceeded the maximum amount that can be stored. If it is determined that the image buffer data 409 has exceeded the maximum amount that can be stored, the program proceeds to step S906. On the other hand, if the image buffer data 409 has not exceeded the maximum amount that can be stored, the program ends the process.

[0075] The criteria for determination in step S905 may be a method of setting a maximum number of images in advance, a method of using the number of pieces of data corresponding to the delay time with the client measured by the delay time measurement program 404, a data capacity that can be stored in the secondary storage device 204 of the camera 102, etc. Any criteria or methods other than those mentioned above may also be used.

[0076] Next, in step S906, the difference detection program 801 deletes the oldest frame stored in the image buffer data 409, and updates the image buffer data 409. In this way, in the second embodiment, the difference detection program 801 also functions as an update unit that updates the image buffer data 409 by deleting or saving a target frame based on the difference between frames.

[0077] 10 is a diagram showing the relationship between an image (frame) captured by the camera 102 at a certain time t5, an image stored as image buffer data 409, and an image displayed on the client device 103 at the same time. In the example shown in FIG. 5, the delay on the network from the camera 102 to the client device 103 and from the client device 103 to the camera 102 is t n -t n-2 This indicates that there are seconds remaining.

[0078] A camera captured image 601, a client display image 602, a user designated area 603, and a camera designated area 604 shown in FIG. 10 are similar to those in the first embodiment, and therefore description thereof will be omitted.

[0079] The deleted image buffer data 1001 is a frame that has been determined by the difference detection program 801 to have a small difference (less than a predetermined threshold) in step S902 described above and has been deleted from the secondary storage device 204 as the image buffer data 409.

[0080] 6. However, in the second embodiment, an image with a small difference between the previous and next frames is not stored as the image buffer data 409. In the second embodiment, the network delay is t n -t n-4 There are seconds, but t n -t n-4 In the second embodiment, the image buffer data 409 from t seconds ago is not used. n -t n-4 t with small difference from seconds ago n -t n-2 The image buffer data from the previous seconds shall be used.

[0081] As described above, according to the second embodiment, by performing a process of deleting the image buffer data 409 depending on the presence or absence of an inter-frame difference, it becomes possible to store the image buffer data 409 for a longer period of time. This makes the system including the camera 102 of the second embodiment effective even between the client device 103 and the system having a large network delay.

[0082] <Example 3> Hereinafter, a system configuration according to the third embodiment will be described with reference to Fig. 11 to Fig. 15. The third embodiment is another embodiment in which the MEC (Mobile Edge Computing) server 1102 in the first or second embodiment is included as a part of the configuration. Note that in the third embodiment, the description of the same configuration and contents as the camera 102 and the client device 103 in the first or second embodiment will be omitted.

[0083] In the third embodiment, 5G (fifth generation mobile communication system) is assumed, and a MEC server (WEB server) 1102 is used as part of the configuration, and at least a part of the processing performed by the camera 102 in the first and second embodiments is performed by the MEC server 1102.

[0084] 11 is a configuration diagram of the entire system in Example 3. In Example 3, the camera 102 is communicably connected to the MEC server 1102 via a 5G network 1101, and the MEC server 1102 configures a system (imaging system) with the client device 103. As a result, the camera 102, the MEC server 1102, and the client device 103 are communicably connected to each other.

[0085] The 5G network 1101 is a fifth-generation mobile communication system. Here, the 5G network 1101 may be a network service provided by a telecommunications carrier or a network not connected to the Internet, called local 5G.

[0086] The MEC server 1102 is a server characterized by edge computing processing, and by executing the processing performed by the camera 102 on the MEC server 1102, it is possible to reduce the processing load of the camera 102 while reducing network latency.

[0087] 12 is a configuration diagram of the camera 102 in the embodiment 3. The camera 102 in the embodiment 3 differs from the camera 102 in the embodiments 1 and 2 in that the camera 102 does not have an image recognition unit 208. The other configurations are similar to those of the camera 102 in the embodiments 1 and 2, and therefore descriptions thereof will be omitted.

[0088] 13 is a configuration diagram of the MEC server 1102 in the third embodiment. The MEC server 1102 has an internal bus 1301 configured therein. The MEC server 1102 is configured to have a CPU 1302, a primary storage device 1303, a secondary storage device 1304, an image recognition unit 1305, and a network I / F 1306, which are connected to the internal bus 1301.

[0089] The CPU 1302 is a central processing unit, and performs overall control of the operations in the MEC server 1102 by executing a control program stored in the primary storage device 1303 .

[0090] The primary storage device 1303 is, for example, a writable high-speed storage device such as a RAM. For example, the OS (operating system), various programs, and various data are loaded into the primary storage device 1303, and the primary storage device 1303 is also used as a working area when the CPU 1302 executes the OS and various programs.

[0091] The secondary storage device 1304 is a non-volatile storage device such as a HDD, a flash memory, or an SD card. The secondary storage device 204 may be configured to be removable. The secondary storage device 1304 is used as a permanent storage area for the OS, various programs, and various data, and is also used as a short-term storage area for various data.

[0092] The image recognition unit 1305 performs image recognition on the image acquired from the camera 102, and instructs the PTZ drive unit 207 of the camera 102 to perform PTZ control to fit a specified target within the angle of view. Here, the MEC server 1102 communicates using a 5G network, making it possible to communicate at a higher speed than in the first and second embodiments.

[0093] The network I / F 1306 is an interface for connecting to the network 101 and the 5G network 1101. Network communication from the client device 103 is transmitted to various programs via the network I / F 101. In addition, network communication with the camera 102 is transmitted to various programs via the 5G network 1101.

[0094] 14 is a configuration diagram showing an example of a program of the camera 102 in the third embodiment. Specifically, it is a diagram showing an example of various programs and various data stored in the primary storage device 203 of the camera 102 in the third embodiment. Here, since various functions are performed on the MEC server 1102, the system is configured with an OS 401, an image capturing program 402, and a communication control program 403. These components are similar to those in the first embodiment, and therefore description thereof will be omitted.

[0095] 15 is a configuration diagram showing an example of a program of the MEC server 1102 in the embodiment 3. Specifically, it is a diagram showing an example of various programs and various data stored in the primary storage device 1303 and the secondary storage device 1304 of the MEC server 1102.

[0096] An OS 1501, a communication control program 1502, a delay time measurement program 404, an image recognition program 405, a target determination program 406, and a PTZ control program 407 are loaded into the primary storage device 1303. The secondary storage device 1304 stores delay time information 408, image buffer data 409, and target image data 410.

[0097] The OS 1501 is a basic program for controlling the entire MEC server 1102. Here, the positions (addresses) and sizes in the primary storage device 1303 of the various programs loaded into the primary storage device 1303 are managed by the OS 1501. The various programs referred to here are the OS 1501, the communication control program 1502, the delay time measurement program 404, the image recognition program 405, the target determination program 406, and the PTZ control program 407.

[0098] The communication control program 1502 is a program for controlling communication with the camera 102 from the network I / F 1306 through the network 1101. Furthermore, the communication control program 1502 is also a program for controlling communication with the client device 103 from the network I / F 1306 through the network 101. One example of the content of communication is sending an acquisition request for an image captured by the imaging unit 206 of the camera 102, or sending a PTZ control request to the PTZ driving unit 207 of the camera 102. Note that requests sent to the camera 102 and the client device 103 may use a general communication protocol such as http (hypertext transfer protocol).

[0099] The MEC server 1102 can identify and track a tracking target based on an area designated by a user by performing the same processing as that performed by the camera 102 in the first embodiment based on the various programs loaded in the primary storage device 1303 described above. Also, a difference detection program 801 may be added to the primary storage device 1303 of the MEC server 1102 to perform the same processing as that performed in the second embodiment.

[0100] As described above, according to the third embodiment, various processes that were previously performed by the camera 102 are realized on the MEC server 1102 instead of the camera 102, thereby achieving the same effect as the first embodiment while reducing network latency, and making it possible to reduce the processing load on the camera 102.

[0101] The functions of the embodiments including the above examples can also be realized by the following configuration. That is, the program code for performing the processing of the above examples is distributed to a system or device, and the computer (or CPU or MPU) of the system or device executes the program code. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the storage medium storing the program code also realizes the functions of the present embodiment.

[0102] Also, the program code for realizing the functions of this embodiment may be executed by one computer (or CPU, MPU), or may be executed by multiple computers working together. Furthermore, the program code may be executed by a computer, or hardware such as a circuit for realizing the functions of the program code may be provided. Alternatively, part of the program code may be realized by hardware, and the remaining part may be executed by a computer.

[0103] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0104] The disclosure of the present embodiment includes the following configurations, methods, programs, and systems.

[0105] (Configuration 1) An imaging device including an imaging means for capturing an image, A transmission means for transmitting the captured image to a client device; A receiving means for receiving a signal for setting a designated area in a tracking target from a client device; an acquisition means for acquiring a past frame image from a plurality of frame images included in a video based on a delay time in communication between the imaging device and the client device; A means for identifying a tracking target from a designated area in a past frame image; and a viewing angle changing means for changing the viewing angle of the imaging device in order to track the tracking target when the tracking target identified by the identifying means is present in the latest frame image among the plurality of frame images included in the video. 1. An imaging device comprising:

[0106] (Configuration 2) The imaging device according to configuration 1, characterized in that the delay time is the sum of the time it takes for the image to arrive at the client device and the time it takes for the imaging device to receive a signal specifying the designated area using the arrived image.

[0107] (Configuration 3) 3. The imaging device according to configuration 1 or 2, further comprising a first calculation means for calculating a movement trajectory of the tracking target from a past frame image to the latest frame image when the tracking target identified by the identification means is not present in the latest frame image among a plurality of frame images included in the video.

[0108] (Configuration 4) The imaging device according to configuration 3, further comprising a second calculation means for estimating a current position of the tracking target based on the movement trajectory, and calculating a movement direction of the tracking target based on the estimated position of the tracking target.

[0109] (Configuration 5) 5. The imaging device according to configuration 4, wherein the angle of view change means changes the angle of view of the imaging device based on the moving direction calculated by the second calculation means.

[0110] (Configuration 6) 6. The imaging device according to any one of configurations 1 to 5, further comprising a detection means for detecting a change between a plurality of frame images.

[0111] (Configuration 7) A storage means for storing a plurality of frame images, 7. The imaging device according to configuration 6, wherein the detection means compares a target frame image with the latest frame image stored in the storage means, and detects a difference therebetween.

[0112] (Configuration 8) The imaging device described in configuration 7 is characterized in that it has an update means for deleting the latest frame image from the storage means when the difference between a target frame image and the latest frame image is less than a predetermined threshold, and for storing the target frame image in the storage means when the difference is equal to or greater than the predetermined threshold.

[0113] (Configuration 9) A measuring means for measuring a delay time is provided, 9. The imaging device according to configuration 7 or 8, wherein the measuring means stores the delay time in a storage means.

[0114] (Configuration 10) A method for controlling an imaging device that captures video, comprising: Transmitting the captured image to a client device; receiving a signal from a client device to set a designated area for a tracking target; acquiring a past frame image from among a plurality of frame images included in the video based on a delay time in communication between the imaging device and the client device; Identify the tracking target from a specified area in a past frame image, When the identified tracking target is present in the latest frame image among a plurality of frame images included in the video, the angle of view of the imaging device is changed in order to track the tracking target. 23. A method for controlling an imaging apparatus comprising the steps of:

[0115] (Configuration 11) A program for causing a computer to execute a control method for an imaging device that captures video, Transmitting the captured image to a client device; receiving a signal from a client device to set a designated area for a tracking target; acquiring a past frame image from among a plurality of frame images included in the video based on a delay time in communication between the imaging device and the client device; Identify the tracking target from a specified area in a past frame image, When the identified tracking target is present in the latest frame image among a plurality of frame images included in the video, the angle of view of the imaging device is changed in order to track the tracking target. A program characterized by:

[0116] (Configuration 12) An imaging system including an imaging device and a WEB server communicably connected to the imaging device, The imaging device An imaging means for capturing an image, The web server is A transmitting means for transmitting the captured image to a client device; A receiving means for receiving a signal for setting a designated area in a tracking target from a client device; an acquisition means for acquiring a past frame image from a plurality of frame images included in a video based on a delay time in communication between the imaging device and the client device; A means for identifying a tracking target from a designated area in a past frame image; and a viewing angle changing means for changing the viewing angle of the imaging device in order to track the tracking target when the tracking target identified by the identifying means is present in the latest frame image among the plurality of frame images included in the video. 1. An imaging system comprising:

[0117] (Configuration 13) 13. The imaging system according to configuration 12, wherein the WEB server is configured as a MEC (Mobile Edge Computing) server.

[0118] (Configuration 14) The imaging system further includes a client device; The imaging system according to configuration 12 or 13, characterized in that the imaging device is communicatively connected to the WEB server, and the client device is communicatively connected to the WEB server, such that the imaging device, the WEB server, and the client device are each communicatively connected. [Explanation of symbols]

[0119] 101 Network 102 Camera 103 Client device 601 Camera Images 602 Client Display Image 603 User-specified area 604 Camera Specified Area 605 Tracking target area

Claims

1. An imaging means for imaging a subject; a transmitting means for transmitting the captured image to a client device; a receiving means for receiving a signal for setting a designated area in a tracking target from the client device; an acquisition means for acquiring a past frame image from a plurality of frame images included in the video based on a delay time in communication between the imaging device and the client device; an identification means for identifying the tracking target from the designated area in the past frame image; and a view angle changing means for changing a view angle of the imaging device in order to track the tracking target when the identified tracking target is present in the latest frame image among a plurality of frame images included in the video. An imaging device characterized by:

2. 2. The imaging device according to claim 1, wherein the delay time is the sum of the time until the image arrives at the client device and the time until the imaging device receives a signal specifying the specified area using the image that has arrived.

3. An imaging device as described in Claim 1, characterized in that it has a first calculation means for calculating the movement trajectory of the tracking target from the past frame image to the latest frame image when the identified tracking target is not present in the latest frame image among the multiple frame images included in the video.

4. 4. The imaging device according to claim 3, further comprising a second calculation means for estimating a current position of the tracking target based on the movement trajectory, and calculating a movement direction of the tracking target based on the estimated position of the tracking target.

5. 5. The imaging device according to claim 4, wherein the angle-of-view changing means changes the angle of view of the imaging device based on the movement direction calculated by the second calculation means.

6. 2. The imaging apparatus according to claim 1, further comprising a detection means for detecting a change between the plurality of frame images.

7. a storage means for storing the plurality of frame images; 7. The imaging device according to claim 6, wherein said detecting means compares a target frame image with the latest frame image stored in said storing means, and detects a difference.

8. The imaging device according to claim 7, further comprising an update means for deleting the latest frame image from the storage means if the difference between the target frame image and the latest frame image is less than a predetermined threshold, and for storing the target frame image in the storage means if the difference is greater than or equal to the predetermined threshold.

9. a measuring means for measuring the delay time, 8. The imaging device according to claim 7, wherein the measuring means stores the delay time in the storage means.

10. An image of a subject is captured, Transmitting the captured image to a client device; receiving a signal from the client device to set a designated area in a tracking target; acquiring a past frame image from a plurality of frame images included in the video based on a delay time in communication between the imaging device and the client device; Identifying the tracking target from the specified area in the past frame image; If the identified tracking target is present in the latest frame image among a plurality of frame images included in the video, changing the angle of view of the imaging device to track the tracking target.

10. A method for controlling an imaging device comprising:

11. A program for causing a computer to execute a control method for an imaging device, comprising: Photograph the subject, Transmitting the captured image to a client device; receiving a signal from the client device to set a designated area in a tracking target; acquiring a past frame image from a plurality of frame images included in the video based on a delay time in communication between the imaging device and the client device; Identifying the tracking target from the specified area in the past frame image; If the identified tracking target is present in the latest frame image among a plurality of frame images included in the video, changing the angle of view of the imaging device to track the tracking target. A program characterized by:

12. An imaging system including an imaging device and a web server communicably connected to the imaging device, The imaging device is an imaging means for capturing an image of a subject and generating an image; The web server a transmitting means for transmitting the image acquired from the imaging device to a client device; a receiving means for receiving a signal for setting a designated area in a tracking target from the client device; an acquisition means for acquiring a past frame image from a plurality of frame images included in the video based on a delay time in communication between the imaging device and the client device; an identification means for identifying the tracking target from the designated area in the past frame image; and a field of view changing means for changing a field of view of the imaging device in order to track the tracking target when the tracking target identified by the identification means is present in the latest frame image among a plurality of frame images included in the video. An imaging system characterized by:

13. 13. The imaging system according to claim 12, wherein the web server is configured as a MEC (Mobile Edge Computing) server.

14. the imaging system further includes the client device; The imaging system according to claim 12, characterized in that the imaging device is communicatively connected to the web server, and the client device is communicatively connected to the web server, thereby communicatively connecting the imaging device, the web server, and the client device.

Images