Image processing device, image processing system, and image processing method

Abstract

The purpose of the present invention is to provide an image processing means capable of performing fast tracking processing on a detection target in wide area monitoring.　Thus, the present invention is configured as an image processing device including: a video acquisition unit (201) for acquiring a first image; an object detection processing unit (202) which executes predetermined object detection processing for the first image, specifies a target object in the first image, and generates a first object detection result indicating the position of the target object on an image; a tracking processing unit (203) which acquires a target area image including the target object on the basis of the first object detection result, generates a resized image by executing resizing processing, which is for converting the target area image to be of a predetermined size, executes predetermined object detection processing for the resized image, and generates a second object detection result indicating at least the position of the target object on an image; and an integration processing unit (204) which integrates the first object detection result with the second object detection result, thereby generating a final object detection result.

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing system, and an image processing method. [Background technology]

[0002] 2. Description of the Related Art Conventionally, surveillance systems using surveillance cameras capable of capturing images of suspicious objects or persons have been provided as part of crime prevention measures in various locations such as stores, streets, and parking lots.

[0003] When using cameras in a surveillance system aimed at monitoring a wide area, a known method is to use a camera equipped with PTZ (Pan-Tilt-Zoom) functionality to patrol the surveillance area and, when it detects a surveillance target such as an intruder or an intruding vehicle, to track the discovered target.

[0004] In addition, in a system that simultaneously captures and monitors a wide surveillance area, there is a method of expanding the area where the intrusion is detected and tracking the target when an intrusion is detected. In particular, in recent years, as the resolution of cameras has increased, it has become more effective to capture a wide area at once using a camera that can obtain wide-angle and high-resolution images, and to expand the area when an intrusion is detected. Methods for enlarging the area in which an intrusion has been detected include digital zoom, which uses image processing to electronically enlarge and display a specific area, and optical zoom, which optically enlarges and displays the area using a lens or the like. In recent years, it has become possible to use image analysis technology to identify intruders and vehicles, etc. Therefore, once an intrusion is detected, it is possible to enlarge the detected person or vehicle and automatically track it.

[0005] An example of a monitoring system is disclosed in Japanese Patent Application Laid-Open No. 2019-124986 (Patent Document 1). Patent Document 1 discloses the following technology: "The obstacle detection system comprises an imaging means (101) for imaging a monitored area, such as a road, at a single or multiple angles of view, and an object extraction means (201) for extracting the area and pixel values ​​of an object occurring in the monitored area from the image captured by the imaging means. The obstacle detection system further comprises an object recognition means (202) for dividing the area and pixel values ​​of the object acquired by the object extraction means (201) into blocks based on criteria set for each angle of view and position in the image, and for identifying the type of object from local features, and an obstacle detection means (204) for detecting the presence or absence of an obstacle occurring in the image from information on the type of object acquired by the object recognition means (202)." [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2019-124986 A Summary of the Invention [Problem to be solved by the invention]

[0007] Patent Document 1 describes a means for detecting an obstacle that occurs within a predetermined monitoring area. However, when detecting and tracking a specific object or event from a large, high-resolution image using conventional means such as those described in Patent Document 1, the amount of calculation required for processing is large. For this reason, when real-time processing is required, it is necessary to use a high-performance computer, etc., which poses issues such as larger equipment size and increased power consumption. In order to improve processing speed, it is possible to reduce the image size before performing detection processing, but in this case, the size (number of pixels) of the detection target becomes small, which can cause it to be overlooked.

[0008] Furthermore, when enlarging and tracking one specific detection target, it is not possible to monitor other areas, so conventional means such as those described in Patent Document 1 are not capable of monitoring multiple detection targets simultaneously.

[0009] Therefore, an object of the present disclosure is to provide an image processing means capable of performing high-speed and high-precision tracking processing of a target object in wide-area surveillance while reducing the load of image processing. [Means for solving the problem]

[0010] In order to solve the above problems, a representative embodiment of the present invention includes an image acquisition unit for acquiring a first image, an object detection processing unit for performing a predetermined object detection process on the first image, identifying a target object in the first image, and generating a first object detection result indicating the position of the target object on the image, a tracking processing unit for acquiring a target area image including the target object based on the first object detection result, performing a resizing process to convert the target area image to a predetermined size to generate a resized image, performing a predetermined object detection process on the resized image, and generating a second object detection result indicating the position of the target object on the image, and an integration processing unit for generating a final object detection result by integrating the first object detection result and the second object detection result. Effect of the Invention

[0011] According to the present disclosure, it is possible to provide an image processing means capable of performing high-speed and high-precision tracking processing of a target object in wide-area surveillance while suppressing the load of image processing. Other objects, configurations and effects will become apparent from the following description of the preferred embodiment of the invention. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates a computer system for implementing an embodiment of the present disclosure. [Diagram 2]FIG. 2 is a diagram illustrating an example of a configuration of an image processing system according to the first embodiment of the present disclosure. [Diagram 3] FIG. 3 is a flowchart illustrating a flow of the object detection process in the first embodiment of the present disclosure. [Figure 4] FIG. 4 is a flowchart illustrating a flow of the tracking process in the first embodiment of the present disclosure. [Diagram 5] FIG. 5 is a flowchart illustrating a flow of the integration process in the first embodiment of the present disclosure. [Figure 6] FIG. 6 is a flowchart illustrating a flow of a display control process in the first embodiment of the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating an example of a display screen in the first embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram illustrating an example of a configuration of an image processing system according to the second embodiment of the present disclosure. [Figure 9] FIG. 9 is a flowchart showing the flow of operations of the image processing system according to the second embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiment. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals. In the following, aspects of the present disclosure may be described by specifying an example such as Example 1 or Example 2, or may be described without specifying an example. An aspect of the present disclosure described by specifying an example is not limited to that example and may be applied to other examples. Also, an aspect of the present disclosure described without specifying an example may be applied to any example such as Example 1 or Example 2.

[0014] As described above, when detecting and tracking a specific object or event from a large, high-resolution image using conventional means, the amount of calculation required for the detection process is large. For this reason, when real-time processing is required, it is necessary to use a high-performance computer, etc., which poses problems such as an increase in the size of the device and an increase in power consumption.

[0015] Therefore, in the present disclosure, only the object detection process is performed on high-resolution images, and the subsequent tracking process is performed on lower-resolution images, thereby enabling high-speed, high-precision tracking of target objects in wide-area surveillance while reducing the overall processing load. This makes it possible to provide highly accurate detection results in real time, even in situations where real-time detection and tracking is required, such as when the target of detection is moving at high speed. Furthermore, by reducing the processing load, image processing in the embodiments of the present disclosure can be implemented on devices with limited power, such as drones.

[0016] First, referring to Fig. 1, a computer system 100 for implementing the embodiments of the present disclosure will be described. The mechanisms and devices of the various embodiments disclosed herein may be applied to any suitable computing system. The main components of the computer system 100 include a processor 102, a memory 104, a terminal interface 112, a storage interface 113, an I / O (input / output) device interface 114, and a network interface 115. These components may be interconnected via a memory bus 106, an I / O bus 108, a bus interface unit 109, and an I / O bus interface unit 110.

[0017] Computer system 100 may include one or more general purpose programmable central processing units (CPUs) 102A and 102B, collectively referred to as processors 102. In some embodiments, computer system 100 may include multiple processors, while in other embodiments, computer system 100 may be a single CPU system. Each processor 102 executes instructions stored in memory 104 and may include an on-board cache. Processors 102 may also include processors capable of high speed computation, such as GPUs, FPGAs, DSPs, ASICs, and the like.

[0018] In one embodiment, memory 104 may include random access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing data and programs. Memory 104 may store all or a portion of the programs, modules, and data structures that implement the functions described herein. For example, memory 104 may store an image processing application 150. In one embodiment, image processing application 150 may include instructions or descriptions for executing on processor 102 the functions described below.

[0019] In some embodiments, image processing application 150 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, image processing application 150 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with bus interface unit 109, processor 102, or other hardware of computer system 100.

[0020] Computer system 100 may include a bus interface unit 109 that provides communication between processor 102, memory 104, display system 124, and I / O bus interface unit 110. I / O bus interface unit 110 may couple to an I / O bus 108 for transferring data to and from various I / O units. I / O bus interface unit 110 may communicate via I / O bus 108 with multiple I / O interface units 112, 113, 114, and 115, also known as I / O processors (IOPs) or I / O adapters (IOAs).

[0021] Display system 124 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both data to display device 126. Computer system 100 may also include one or more sensors or other devices configured to collect data and provide the data to processor 102.

[0022] For example, computer system 100 may include biometric sensors to collect heart rate data, stress level data, etc., environmental sensors to collect humidity data, temperature data, pressure data, etc., and motion sensors to collect acceleration data, movement data, etc. Other types of sensors may also be used. Display system 124 may be connected to a display device 126, such as a standalone display screen, a television, a tablet, or a handheld device.

[0023] The I / O interface unit provides the ability to communicate with various storage or I / O devices. For example, the terminal interface unit 112 may be attached to user I / O devices 116, such as user output devices, such as a video display, a television with speakers, and user input devices, such as a keyboard, a mouse, a keypad, a touchpad, a trackball, buttons, a light pen, or other pointing device. A user may use a user interface to enter input data or instructions to the user I / O devices 116 and the computer system 100, and receive output data from the computer system 100, by manipulating the user input devices. The user interface may be displayed on a display, played through speakers, or printed via a printer, for example, via the user I / O devices 116.

[0024] Storage interface 113 may be attached to one or more disk drives or direct access storage device 117 (usually a magnetic disk drive storage device, but may also be an array of disk drives or other storage devices configured to appear as a single disk drive). In one embodiment, storage device 117 may be implemented as any secondary storage device. Contents of memory 104 may be stored in storage device 117 and retrieved from storage device 117 as needed. I / O device interface 114 may provide an interface to other I / O devices such as printers, fax machines, etc. Network interface 115 may provide a communications path to allow computer system 100 and other devices to communicate with each other. This communications path may be, for example, network 130.

[0025] In some embodiments, computer system 100 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. In other embodiments, computer system 100 may be a desktop computer, a portable computer, a laptop, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device. EXAMPLES

[0026] Next, the configuration of the image processing system in the first embodiment of the present disclosure will be described with reference to FIG.

[0027] Fig. 2 is a diagram showing an example of the configuration of an image processing system 200 in the first embodiment of the present disclosure. The image processing system 200 in the first embodiment of the present disclosure is a system for performing high-speed tracking processing of a target object in wide-area surveillance, and as shown in Fig. 2, is mainly composed of an image capturing device 201 and an image processing device 210. The image capturing device 201 and the image processing device 210 are connected to each other so as to be able to communicate with each other via a communication network 206 such as the Internet.

[0028] The image capturing device 201 is a functional unit configured to capture an image of a predetermined environment and capture image data showing the environment. The image capturing device 201 may be, for example, a normal camera with a fixed angle of view, a camera with adjustment functions such as pan, tilt, and zoom, or a rotating camera capable of 360-degree rotation. The image capturing device 201 may be installed in advance at a position where the predetermined environment can be captured, or may be mounted on a moving object such as a drone, as described later. The video data acquired by the video acquisition device 201 is an image sequence consisting of a plurality of consecutive image frames. This video data may also be a high-resolution video. Here, a "high-resolution" image means an image that satisfies a first pixel count standard. This first pixel count standard is a threshold value that specifies a specific lower limit of the pixel count, and may be, for example, a pixel count of 1920 pixels x 1080 pixels (FHD) or more, a pixel count of 4K (4096 pixels x 2160 pixels or 3840 pixels x 2160 pixels) or more, a pixel count of 8K (7680 pixels x 4320 pixels) or more, etc. Note that the installation locations and number of the image capturing devices 201 are not particularly limited in this disclosure and may be appropriately determined depending on the purpose of monitoring, etc. Also, while a case where the image capturing device 201 is a device connected to the image processing device 210 via the communication network 206 is shown as an example here, the present disclosure is not limited to this, and the image capturing device 201 may be implemented as an image processing unit in the image processing device 210.

[0029] The image processing device 210 is a device that executes the image processing means in the embodiment of the present disclosure after receiving the video data acquired by the video acquisition device 201 via a communication network. As shown in FIG. 2, the image processing device 210 includes an object detection processing unit 202, a tracking processing unit 203, an integration processing unit 204, and a display control unit 205.

[0030] The object detection processing unit 202 is a functional unit that performs a predetermined object detection process on a specific image frame (hereinafter referred to as the "first image") in the video data acquired by the video acquisition device 201, thereby identifying a target object in the first image and generating a first object detection result that indicates at least the position of the target object on the image. When the first image is an image frame from high-resolution video data, it goes without saying that the image has a high resolution similar to the video data. Generally, the higher the image resolution, the slower the object detection process becomes, but in the present disclosure, the object detection process may be capable of processing video data at about 1 to 3 FPS. The details of the processing by the object detection processing unit 202 will be described later, and therefore will not be described here.

[0031] The target object here means an object to be detected in an image. The target object may be appropriately set by an administrator of the image processing system 200, for example, when setting up the object detection process. For example, the target object here may be any object, such as a person with a certain characteristic (a woman wearing a red hat, a man holding a gun), an animal, a car, a building, etc.

[0032] The tracking processing unit 203 is a functional unit that acquires a target area image including the detected target object based on the first object detection result generated by the object detection processing unit 202, and generates a resized image by executing a resizing process for converting the target area image into a predetermined size. Thereafter, the tracking processing unit 203 executes a predetermined object detection process on the resized image, and generates a second object detection result indicating at least the position of the target object on the image. The resized image here is an image that falls below a second pixel count standard. The second pixel count standard is a threshold value that specifies a specific upper pixel count limit, and may be, for example, 1920 pixels x 1080 pixels (FHD) or less, 640 pixels x 480 pixels or less, 320 pixels x 240 pixels or less, 50% or less of the first image, etc. In this way, the resized image has a lower resolution than the first image. Therefore, the object detection process performed by the tracking processing unit 203 on the resized image has a lower processing load and is faster (for example, 10 FPS or higher) than the object detection process performed by the object detection processing unit 202 on the first image. The details of the processing by the tracking processing unit 203 will be described later, and therefore will not be described here.

[0033] The target area image here is an image acquired centered around the target object detected by the object detection processing unit 202. In one aspect of the present disclosure, the tracking processing unit 203 may extract the target area image by cutting it out from the first image based on the first object detection result. In another aspect of the present disclosure, the tracking processing unit 203 determines the shooting conditions (e.g., pan, tilt, and zoom settings for shooting the target object clearly and near the center of the image) for shooting a target area image including the target object based on the first object detection result, and transmits the determined shooting conditions to the image capturing device 201. After that, the image capturing device 201 transmits the image acquired by shooting according to these shooting conditions to the tracking processing unit as a target area image. This makes it possible to acquire the target area image without processing the first image.

[0034] The integration processing unit 204 is a functional unit that generates a final object detection result by integrating the first object detection result from the object detection processing unit 202 and the second object detection result from the tracking processing unit 203. The final object detection result here is information obtained by integrating the first object detection result and the second object detection result, and therefore is a result that indicates the position of the target object more accurately than the first object detection result and the second object detection result. For example, here, the integration processing unit 204 may generate a final object detection result indicating an estimated position of the target object by performing a so-called IoU (Intersection Over Union) process using the first object detection result and the second object detection result. The details of the processing by the integration processing unit 204 will be described later, and therefore will not be described here.

[0035] The display control unit 205 is a functional unit for displaying the final object detection result generated by the integration processing unit 204. The details of the processing performed by the display control unit 205 will be described later, and therefore will not be described here.

[0036] The various functional units included in the image processing device 210 described above may be implemented as software modules of an image processing application 150 stored in the memory of the computer system 100 shown in FIG. 1, for example. On the other hand, various functional units included in the image processing device 210 may be implemented on different computers. In this case, since the object detection processing unit 202 is an image analysis unit for detecting a target object from a large-sized image such as 4K or FHD, it is desirable that the object detection processing unit 202 is implemented on a computer with higher performance than the tracking processing unit. As an example, a configuration in which the object detection processing unit 202 is implemented on a high-performance computer such as a cloud, and the tracking processing unit is implemented on a moving body such as a drone, etc. can be considered.

[0037] According to the image processing system 200 described above, only the object detection process is performed on a high-resolution image (first image), and the subsequent tracking process is performed on a lower-resolution image (resized image), thereby enabling high-speed and high-precision tracking of target objects in wide-area surveillance while reducing the load of image processing.

[0038] Next, the object detection process in the first embodiment of the present disclosure will be described with reference to FIG.

[0039] 3 is a flowchart showing the flow of an object detection process 300 in the first embodiment of the present disclosure. The object detection process 300 shown in FIG 3 is a process for determining a target object in a high-resolution image, and is executed by, for example, the object detection processing unit 202 shown in FIG 2.

[0040] First, in step S311, the object detection processing unit 202 acquires a specific image frame (hereinafter referred to as a "first image") from the video data acquired by the video acquisition device 201. Here, the object detection processing unit 202 may acquire the first frame in the video data transmitted in real time from the video acquisition device 201 as the first image.

[0041] Next, in step S312, the object detection processing unit 202 executes a predetermined object detection process on the first image acquired in step S311. The object detection process here may include any existing object detection processing means, such as Viola-Jones object detection framework based on Haar features, SIFT (Scale-invariant feature transform), HOG (Histogram of oriented gradients), R-CNN (Region-based convolutional neural network), Fast R-CNN, Faster R-CNN, Cascade R-CNN, SSD (Single Shot MultiBox Detector), YOLO (You Only Look Once), RefineDet (Single-Shot Refinement Neural Network for Object Detection), Retina-Net, and Deformable convolutional networks. As described above, the object detection process executed here is performed on high-resolution images, and therefore is performed at a slow speed compared to the frame rate of the image capture device 201. However, since a wide range is captured, the target object will not be missed.

[0042] By performing the object detection process, the object detection processing unit 202 can generate a first object detection result for the first image indicating the position (coordinates on the image) of each detected target object and the class of the target object.

[0043] Next, in step S313, the object detection processing unit 202 transmits the first object detection result generated in step S312 to the tracking processing unit 203 and the integration processing unit 204. After that, the tracking process 400 shown in FIG. After the processing for the first image is completed, the process returns to step S311, and processing for the next image frame in the video data (i.e., the image frame following the first image frame) is started. In this manner, each frame in the video data is processed sequentially, and an object detection result for each frame is generated.

[0044] Next, a tracking process in the first embodiment of the present disclosure will be described with reference to FIG.

[0045] 4 is a flowchart showing the flow of a tracking process 400 in the first embodiment of the present disclosure. The tracking process 400 shown in FIG 4 is a process for tracking a target object, and is executed by, for example, the tracking processing unit 203 shown in FIG 2.

[0046] First, in step S421, upon receiving the first object detection result from the object detection processing unit, the tracking processing unit 203 starts tracking processing. In addition, in the first object detection result, when multiple target objects are identified, the tracking processing unit 203 starts a tracking process for each of the identified target objects. However, for convenience of explanation, the tracking process for one target object will be explained here.

[0047] Next, in step S422, the tracking processing unit 203 acquires an image frame in which the target object is specified (hereinafter referred to as a "first image") based on the first object detection result.

[0048] Next, in step S423, the tracking processing unit 203 determines a target area including the target object based on the first object detection result, and acquires an image of the target area. Here, the target area means an area on the image that at least shows the target object. In addition, it is desirable to set this target area larger than the size of the target object. For example, this target area may be three times larger in both length and width than the target object.

[0049] In one aspect of the present disclosure, the tracking processing unit 203 may extract a target area image by cutting it out from the first image based on the coordinates on the image of the target object indicated in the first object detection result. In another aspect of the present disclosure, the tracking processing unit 203 determines the shooting conditions (e.g., pan, tilt, and zoom settings for shooting the target object clearly and near the center of the image) for shooting a target area image including the target object based on the first object detection result, and transmits the determined shooting conditions to the image capturing device 201. After that, the image capturing device 201 transmits the image acquired by shooting according to these shooting conditions to the tracking processing unit as a target area image. This makes it possible to acquire the target area image without processing the first image.

[0050] After acquiring the target area image, the tracking processing unit 203 performs a resizing process to convert the acquired target area image into a predetermined size, and generates a resized image. Here, the tracking processing unit 203 may perform the resizing process by enlarging or reducing the target area image. In addition, the size of the resized image is not particularly limited, and may be set appropriately in consideration of the accuracy and speed of the object detection process. As an example, the tracking processing unit 203 may resize the target area image to VGA (Video Graphics Array; 640 pixels x 480 pixels) or QVGA (Quarter Video Graphics Array; 320 pixels x 240), etc. As described in this disclosure, converting the target area image into a resized image with a lower resolution can reduce the tracking process load and processing time.

[0051] Next, in step S424, the tracking processing unit 203 executes a predetermined object detection process on the resized image generated in step S423. The object detection process here may be the same as the object detection process used in the object detection process 300 described above, for example, or may be a different object detection process. For example, the tracking processing unit 203 may perform object tracking by matching feature points between frames (the previous frame and the next frame of the first image) using a KLT (Kanade-Lucas-Tomasi) tracker or the like. New feature points are extracted from within the object region, and tracking is continued until the feature points satisfy a tracking end condition, such as disappearance or stationary. By performing such object tracking processing, the trajectory of each feature point moving within the screen can be obtained. By clustering the trajectory of each feature point based on information such as its position, moving direction, and the object region from which the feature point was obtained, multiple clusters representing the movement of an object moving within the video can be obtained. As described above, the resized image has a lower resolution than the first image, so the object detection process performed on the resized image in the tracking process 400 has a lower processing load and is faster than the object detection process performed on the first image in the object detection process 300.

[0052] In this way, by performing object detection processing on the resized image, the tracking processing unit 203 can generate a second object detection result indicating the position on the image (coordinates on the image) of each detected target object and the class of the target object.

[0053] Next, in step S425, the tracking processing unit 203 determines whether or not a target object is detected by the object detection process. If a target object is detected, the process proceeds to step S426. On the other hand, if a target object is not detected, the process proceeds to step S427.

[0054] Next, in step S426, the tracking processing unit 203 transmits the second object detection result generated in step S424 to the above-mentioned integration processing unit 204. Then, the integration process 500 shown in Fig. 5 is started, and the process returns to step S422 to start processing the next image frame in the video data.

[0055] Next, in step S427, the tracking processing unit 203 determines whether the number of frames in which the target object is not detected is equal to or greater than a predetermined number T. If the number of frames in which the target object is not detected is equal to or greater than the predetermined number "T", the tracking processing unit 203 determines that the target object has been lost, and the process ends. On the other hand, if the number of frames in which the target object is not detected is less than the predetermined number "T", the process returns to step S422, and processing of the next image frame in the video data is started.

[0056] Next, the integration process in the first embodiment of the present disclosure will be described with reference to FIG.

[0057] Fig. 5 is a flowchart showing a flow of an integration process 500 in the first embodiment of the present disclosure. The integration process 500 shown in Fig. 5 is a process for integrating a first object detection result generated by the object detection process 300 and a second object detection result generated by the tracking process 400, and is executed by, for example, the integration processing unit 204 shown in Fig. 2.

[0058] As described above, according to the object detection process 300 and the tracking process 400, two object detection results indicating the position of the target object are obtained. However, the position of the target object on the image may differ between the first object detection result and the second object detection result. Therefore, by integrating the first object detection result and the second object detection result into one by the integration process 500 shown in FIG. 5, it is possible to obtain a final object detection result that more reliably indicates the position of the target object on the image.

[0059] First, in step S531, the integration processing unit 204 receives the first object detection result from the object detection processing unit 202.

[0060] Next, in step S532, the integration processing unit 204 receives the second object detection result from the tracking processing unit 203.

[0061] Next, in step S533, the integration processing unit 204 aligns and overlaps the first object detection result and the second object detection result, and determines whether the positions of the target objects in the images overlap. If the areas of the target object in the first object detection result and the second object detection result overlap each other, the integration processing unit 204 determines the position of the target object based on the degree of overlap of the areas of the detected target object and a predetermined IoU threshold, for example using IoU (Intersection Over Union), and generates the final object detection result by integrating the first object detection result and the second object detection result. On the other hand, if the areas of the target object in the first object detection result and the second object detection result do not overlap with each other, the integration processing unit 204 may adopt both the first object detection result and the second object detection result as the final object detection result.

[0062] Next, in step S534, the integration processing unit 204 saves the image frame number in the video data, the position of the target object, the class of the target object, and the detection ID of the target object as the final object detection result generated in step S533 in a specified memory area.

[0063] Next, in step S535, if the integration processing unit 204 determines that there is a new detection in the final object detection result generated in step S533 (that is, if the target object position, the target object class, or the target object detection ID of the final object detection result generated in step S533 is different from the previously saved final object detection result), the process proceeds to step S536 and starts a new tracking process. If it is determined that there is no new detection, the process ends.

[0064] Next, a display control process in the first embodiment of the present disclosure will be described with reference to FIG.

[0065] Fig. 6 is a flowchart showing a flow of a display control process 600 in the first embodiment of the present disclosure. The display control process 600 shown in Fig. 6 is a process for displaying a final object detection result generated by the integration processing unit 204, and is executed by, for example, the display control unit 205 shown in Fig. 2.

[0066] First, in step S637, the display control unit 205 acquires the latest final object detection result from among the final object detection results stored in the storage area in the above-described integration process 500.

[0067] Next, in step S638, the display control unit 205 generates a display screen for displaying the final object detection result acquired in step S637. An example of the display screen is shown in FIG. 7, so its description will be omitted here.

[0068] Next, in step S639, the display control unit 205 outputs the display screen generated in step S638 to a predetermined display device (such as a computer display, or a screen of a smartphone or tablet terminal).

[0069] Next, a display screen in the first embodiment of the present disclosure will be described with reference to FIG.

[0070] 7 is a diagram illustrating an example of a display screen 700 according to the first embodiment of the present disclosure. As described above, the display screen 700 is a screen for displaying a final object detection result generated by the image processing device according to the first embodiment of the present disclosure. More specifically, as shown in FIG. 7, based on the final object detection result generated by the integration processing unit 204, the display control unit 205 generates image 701 in which a rectangle is superimposed on the position where the target object is detected, and images 702, 703, and 704 in which the area around the detected target object is enlarged and a rectangle is superimposed on the position of the target object, and displays them in a tiled manner on the display screen 700.

[0071] Furthermore, when many target objects are detected, the display control unit 205 may display reduced thumbnail images 705 of target objects other than the target objects shown in the images 701 to 704 at the edge of the display screen 700. By selecting a thumbnail image 705, the user can replace the selected thumbnail image 705 with any of the images 701 to 704.

[0072] As described above, according to the image processing means in Example 1 of the present disclosure, only the object detection process is performed on the high-resolution image (first image), and the subsequent tracking process is performed on the lower-resolution image (resized image), thereby enabling high-speed and high-precision tracking of target objects in wide-area surveillance while reducing the load of image processing. EXAMPLES

[0073] Next, an image processing system according to a second embodiment of the present disclosure will be described with reference to FIGS.

[0074] In the above-mentioned first embodiment, the image capturing device 201 of the present disclosure is described as a surveillance camera or the like installed in a specific location, but the present disclosure is not limited thereto, and the image capturing device 201 can also be configured to be mounted on a moving object such as a drone. Therefore, in the second embodiment of the present disclosure, an image processing system 800 will be described in which the image capturing device 201 is mounted on a drone and part of the image processing is executed on the drone side. However, the present disclosure is not limited to a drone, and the image capturing device 201 may be mounted on, for example, a robot, a car driven by a human, or the like.

[0075] Fig. 8 is a diagram showing an example of the configuration of an image processing system 800 according to the second embodiment of the present disclosure. The configuration of the image processing system 800 shown in Fig. 8 is substantially similar to the configuration of the image processing system 200 according to the first embodiment, so for convenience of explanation, the explanation of the common parts will be omitted and the explanation will be centered on the differences between the first embodiment and the second embodiment.

[0076] The image processing system 800 in the second embodiment of the present disclosure is a system for performing high-speed tracking processing of a target object in wide-area surveillance, and is mainly composed of a drone 805 and an image processing device 810, as shown in Fig. 8. The drone 805 and the image processing device 810 are connected to each other by wireless communication via a communication network 206 such as the Internet. It should be noted that the image processing device 810 here may be implemented, for example, in a terrestrial computer, server device, or the like.

[0077] The drone 805 is an unmanned aerial vehicle that flies using rotors, etc. The drone 805 in the second embodiment of the present disclosure is not particularly limited, and may be any drone as long as it is equipped with a camera (image acquisition unit 820) capable of acquiring high-resolution images, a calculation function (tracking processing unit 203) capable of executing image processing in the embodiment of the present disclosure, and a wireless communication function (not shown) for communicating with the image processing device 810.

[0078] As shown in FIG. 8, the drone 805 includes a tracking processing unit 203, a moving object control unit 815, and an image acquisition unit 820. The tracking processing unit 203 is substantially similar to the tracking processing unit 203 in the first embodiment, and therefore a description thereof will be omitted here. The mobile object control unit 815 is a functional unit for controlling the movement and functions of the drone 805, and may be implemented, for example, as a microcomputer or a SoC (System on a Chip) mounted on the drone 805. The mobile object control unit 815 may control the movement of the drone 805 based on an instruction received from the mobile object management unit 803 of the image processing device 810, for example. The image capture unit 820 is a camera capable of capturing high-resolution images, and is substantially similar to the image capture device 201 in the first embodiment, so a description thereof will be omitted here.

[0079] The image processing device 810 in the second embodiment differs from the image processing device 210 in the first embodiment in that the tracking processing unit 203 is mounted on a drone 805 and includes a moving object management unit 803. The mobile object management unit 803 is a functional unit for generating instructions for controlling the movement of the drone 805 and transmitting the instructions to the drone 805. The mobile object management unit 803 may generate a tracking command for tracking a detected target object based on, for example, an object detection result of the object detection processing unit 202, and transmit the tracking command to the drone 805.

[0080] Next, the flow of operations of the image processing system 800 according to the second embodiment of the present disclosure will be described with reference to FIG.

[0081] FIG. 9 is a flowchart showing an operation flow 900 of the image processing system 800 in the second embodiment of the present disclosure.

[0082] First, in step S905, the image acquisition unit 820 in the drone 805 acquires a specific image frame (hereinafter referred to as the "first image") from the high-resolution video data, and transmits the acquired first image to the image processing device 810 via high-speed, high-capacity wireless communication.

[0083] Next, in step S910, the object detection processing unit 202 of the image processing device 810 executes the above-mentioned object detection process (for example, the object detection process 300 shown in FIG. 3) on the first image received from the drone 805 to identify the target object in the first image and generate a first object detection result indicating at least the position of the target object on the image. Thereafter, the object detection processing unit 202 transmits the generated first object detection result to the mobile object management unit 803.

[0084] Next, in step S915, the moving object management unit 803 of the image processing device 810 creates a tracking command for tracking the detected target object based on the first object detection result received from the object detection processing unit 202. The tracking command here is information requesting the drone 805 to track the specific detected target object. The mobile management unit 803 then transmits the created tracking command to the drone 805.

[0085] Next, in step S920, the mobile object control unit 815 may control the movement of the drone 805 and the shooting conditions (pan, tilt, zoom, etc.) of the image acquisition unit 820 based on the tracking command received from the mobile object management unit 803 of the image processing device 810 so that the target object is captured clearly and near the center of the image.

[0086] Next, in step S925, the tracking processing unit 203 executes the above-mentioned tracking process (for example, tracking process 400 shown in FIG. 4). More specifically, the tracking processing unit 203 acquires a target area image for the target object specified in the tracking command. Here, the tracking processing unit 203 may cut out the target area image from the first image acquired in step S905, or may cut out the target area image from a new image acquired by the video acquisition unit 820. Then, the tracking processing unit 203 generates a resized image by performing a resizing process to convert the target area image to a predetermined size, and then performs a predetermined object detection process on the resized image to generate a second object detection result that indicates at least the position of the target object on the image. In addition, since the resized image here has a lower resolution than the first image, the object detection process performed on the target object image by the tracking processing unit 203 of the drone 805 has a lower processing load and is faster than the object detection process performed on the first image by the object detection processing unit 202 of the image processing device 810. This makes it possible to reduce the amount of calculation of the process performed by the drone 805 and the power consumption of the drone 805.

[0087] Next, in step S930, the mobile unit control unit 815 controls the drone 805 to track the target object based on the second object detection result generated in step S925. As a result, the drone 805 captures the target object while tracking the target object, and transmits the video data thus acquired (e.g., a second image) to the image processing device 810. Thereafter, the image processing device 810 executes the above-mentioned integration process and display process, and also executes the processes from step S910 onwards on the newly acquired second image.

[0088] According to the image processing system in Example 2 of the present disclosure described above, object detection processing for high-resolution images is performed on the image processing device on the ground, and tracking processing is performed on lower-resolution images on the drone side, thereby enabling high-speed tracking processing of target objects in wide-area surveillance while reducing the processing load and power consumption of the drone.

[0089] In this way, according to the image processing means in the embodiment of the present disclosure, by performing only the object detection process on a high-resolution image and then performing the tracking process on a lower resolution image, it is possible to reduce the processing load and improve the processing speed while maintaining the accuracy of the tracking process compared to when both the object detection process and the tracking process are performed on a high-resolution image. This makes it possible to provide highly accurate detection results in real time, even in situations where real-time detection and tracking is required, such as when the target of detection is moving at high speed. Furthermore, by reducing the processing load, image processing in the embodiments of the present disclosure can be implemented on devices with limited power, such as drones.

[0090] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the gist of the present invention. Furthermore, it goes without saying that functional units in the present invention, such as the image acquisition unit, object detection processing unit, tracking processing unit, and integration processing unit, may also have functions other than those described above. [Explanation of symbols]

[0091] 200-800: image processing system, 201: image acquisition device, 202: object detection processing unit, 203: tracking processing unit, 204: integration processing unit, 205: display control unit, 206: communication network, 210-810: image processing device, 803: mobile object management unit, 805: drone, 815: mobile object control unit, 820: image acquisition unit

Claims

1. An image processing device, an image capture unit for capturing a first image; an object detection processing unit that performs a predetermined object detection process on the first image, identifies a target object in the first image, and generates a first object detection result indicating a position of the target object on the image; a tracking processing unit that acquires a target area image including the target object based on the first object detection result, performs a resizing process to convert the target area image to a predetermined size to generate a resized image, performs a predetermined object detection process on the resized image, and generates a second object detection result indicating a position of the target object on the image; an integration processing unit that generates a final object detection result by integrating the first object detection result and the second object detection result; The tracking processing unit is determining a photographing condition for photographing the target area image including the target object based on the first object detection result, and transmitting the determined photographing condition to the image acquisition unit; The image acquisition unit is receiving the photographing conditions, acquiring the target area image by photographing based on the received photographing conditions, and transmitting the acquired target area image to the tracking processing unit; 13. An image processing device comprising:

2. The tracking processing unit is The image processing device according to claim 1 , wherein the target area image is extracted from the first image based on the first object detection result.

3. the first image is an image that satisfies a first pixel count criterion; The resized image is an image that is below a second pixel count criterion.

3. The image processing device according to claim 2, wherein the image processing device is a digital camera.

4. An image processing device, comprising: an image capture unit for capturing a first image; an object detection processing unit that performs a predetermined object detection process on the first image, identifies a target object in the first image, and generates a first object detection result indicating a position of the target object on the image; a tracking processing unit that acquires a target area image including the target object based on the first object detection result, performs a resizing process to convert the target area image to a predetermined size to generate a resized image, performs a predetermined object detection process on the resized image, and generates a second object detection result indicating a position of the target object on the image; an integration processing unit that generates a final object detection result by integrating the first object detection result and the second object detection result; The integration processing unit includes: superimposing the first object detection result and the second object detection result; integrating the first object detection result and the second object detection result based on an overlapping degree between the target object indicated in the first object detection result and the target object indicated in the second object detection result, to generate the final object detection result.

13. An image processing device comprising:

5. The tracking processing unit: extracting the target area image from the first image based on the first object detection result; 5. The image processing device according to claim 4,

6. The first image is an image that satisfies a first pixel count criterion, The resized image is an image that is below a second pixel count criterion.

6. The image processing device according to claim 5,

7. In an image processing system in which a mobile object equipped with an image capture unit for capturing images and an image processing device are connected via a communication network, The image processing device includes: an object detection processing unit that performs a predetermined object detection process on a first image received from the image capture unit device, identifies a target object in the first image, and generates a first object detection result indicating a position of the target object on the image; a moving object instruction unit that generates a tracking command for tracking the target object based on the first object detection result, and transmits the generated tracking command to the moving object; The moving body is a tracking processing unit that acquires a target area image including the target object based on the tracking command, performs a resizing process to convert the target area image to a predetermined size to generate a resized image, performs a predetermined object detection process on the resized image, and generates a second object detection result indicating a position of the target object on the image; a moving object control unit that controls the moving object so as to track the target object based on the second object detection result; an image acquisition unit that acquires a second image showing the target object while tracking the target object and transmits the second image to the image processing device; Including, The tracking processing unit is determining a photographing condition for photographing the target area image including the target object based on the first object detection result, and transmitting the determined photographing condition to the image acquisition unit; The image acquisition unit receives the shooting conditions, acquires the target area image by taking pictures based on the received shooting conditions, and transmits the acquired target area image to the tracking processing unit.

8. The first image is an image that satisfies a first pixel count criterion; The resized image is an image that is below a second pixel count criterion; 8. The image processing system according to claim 7.

9. 1. An image processing method, comprising: acquiring a first image; performing a predetermined object detection process on the first image to identify a target object in the first image and generate a first object detection result indicating a position of the target object on the image; extracting a target area image including the target object from the first image based on the first object detection result; determining a photographing condition for photographing the target area image including the target object based on the first object detection result; acquiring the target area image by performing photography based on the photography conditions; performing a resizing process to convert the target area image to a predetermined size to generate a resized image having a lower resolution than the first image and below a predetermined pixel count threshold; performing a predetermined object detection process on the resized image to generate a second object detection result indicating a position of the target object on the image; a step of superimposing the first object detection result and the second object detection result, and integrating the first object detection result and the second object detection result based on an overlapping degree of the target object indicated in the first object detection result and the target object indicated in the second object detection result, thereby generating a final object detection result; 13. An image processing method comprising:

Images