Object detection system, camera, and object detection method

Abstract

The purpose of the present invention is to facilitate the setting of an appropriate distant area for realizing high-speed, high-accuracy object detection regardless of the camera installation conditions.　An image processing device 120 according to the present disclosure comprises: a function for setting, on the basis of positions and sizes of a plurality of object frames (221, 222) that respectively surround a plurality of persons that are included in a camera image (200) that is captured in advance by an image capture device 110, a distant area that includes an area in which the sizes of the object frames in the image area of the camera image become equal to or less than a threshold value; and a function for generating, on the basis of a camera image that is captured during operation, a reduced image that is a reduction of the camera image and a trimmed image in which the distant area portion is trimmed from the camera image, inputting both the reduced image and the trimmed image into a low-resolution learning model to perform person detection, and synthesizing and outputting the detection result based on the reduced image and the detection result based on the trimmed image.

Description

[Technical Field]

[0001] The present invention relates to an object detection system that detects an object from a camera image of a monitoring area. [Background technology]

[0002] Conventionally, research and development has been conducted on object detection systems that detect targets by analyzing camera images of a monitored area using a learning model for object detection. Object detection can be performed using various methods that utilize AI (Artificial Intelligence) technology.

[0003] In the first conventional method shown in FIG. 1, a high-resolution camera image 11, such as a 4K video image, is input directly to a high-resolution learning model 12 for analysis. The high-resolution learning model 12 outputs a resultant image 13 as an analysis result, in which a detection frame indicating a detected object is added to the input image. The resultant image 13 is displayed on a monitor terminal either directly or after being adjusted for user confirmation. The first conventional method can achieve highly accurate detection, but analysis by the high-resolution learning model 12 takes a long time. Furthermore, the high-resolution learning model 12 requires a significant amount of time for learning.

[0004] In the second conventional method shown in FIG. 2, a high-resolution camera image 21 is reduced to a predetermined size (e.g., VGA size) to generate a reduced image 22, which is then input to a low-resolution learning model 23 for analysis. The low-resolution learning model 23 outputs a resultant image 24 as the analysis result, in which a detection frame indicating the detected object is added to the input image. Based on this resultant image 24, a final output image 25 is generated in which the detection frame is reflected in the original camera image 21 (or a camera image adjusted for user confirmation), and is displayed on a monitor terminal. According to the second conventional method, analysis by the low-resolution learning model 23 can be performed quickly, but image reduction makes it difficult to see small objects in the distance, resulting in a lower detection rate.

[0005] In the third conventional method shown in Fig. 3, a reduced image 32 is generated by reducing a high-resolution camera image 31 to a predetermined size, and a cropped image 33 is generated by cutting out a distant region from the camera image. These images are then input to a low-resolution learning model 34 for analysis. As analysis results, the low-resolution learning model 34 outputs a result image 35 showing the detection result based on the reduced image 32 and a result image 36 showing the detection result based on the cropped image 33. These result images 35 and 36 are combined to generate a final output image 37 in which the detection frame is reflected in the original camera image 37, and the final output image 37 is displayed on a monitor terminal.

[0006] The third conventional method can achieve high-speed and high-precision object detection. When the installation conditions of the camera are fixed, such as in the case of the vehicle-mounted camera disclosed in Patent Document 1, the third conventional method does not pose any particular problems because the far area in the camera image is also fixed. However, when various installation conditions depending on the situation at the site are expected, such as in the case of a surveillance camera, the far area setting differs for each camera. Furthermore, setting an appropriate far area requires specialized knowledge and is a tedious and time-consuming task. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2020-4366 Summary of the Invention [Problem to be solved by the invention]

[0008] The present invention has been made in consideration of the above-described conventional circumstances, and aims to facilitate the setting of an appropriate far-field area to achieve high-speed, high-precision object detection regardless of the camera installation conditions. [Means for solving the problem]

[0009] In order to achieve the above object, an object detection system according to one aspect of the present invention is configured as follows: That is, in an object detection system that detects objects from camera images captured of a surveillance area, before operation begins, the system executes a process of setting a distant region including an area in the image area of ​​the camera image where the size of the object frame is equal to or smaller than a threshold, based on the positions and sizes of multiple object frames that surround each of multiple objects included in the camera image captured in advance, and during operation, the system generates a first image by reducing the captured camera image and a second image by cropping out a portion of the distant region from the camera image, based on the captured camera image, inputs both the first image and the second image into a predetermined learning model to detect objects, and executes a process of combining and outputting the detection results based on the first image and the second image.

[0010] Here, the plurality of object frames may be set by a user's operation on a previously captured camera image, or may be set based on a detection result obtained by inputting a previously captured camera image into a learning model.

[0011] A camera according to another aspect of the present invention is configured as follows: That is, in a camera that photographs a surveillance area to detect objects, the camera has a function of setting a distant region including an area in the image region of the camera image where the size of the object frame is equal to or smaller than a threshold, based on the positions and sizes of multiple object frames that respectively surround multiple objects included in a camera image photographed before the start of operation, and a function of generating a first image by reducing the camera image and a second image by cutting out a portion of the distant region from the camera image, based on a camera image photographed during operation, inputting both the first image and the second image into a predetermined learning model to detect objects, and outputting a combination of the detection results based on the first image and the second image.

[0012] An object detection method according to yet another aspect of the present invention is configured as follows: That is, the object detection method detects objects from camera images taken of a surveillance area, and includes the steps of: before starting operation, setting a distant region including an area in the image region of the camera image where the size of the object frame is equal to or smaller than a threshold based on the positions and sizes of multiple object frames surrounding each of multiple objects included in the camera image taken in advance; and during operation, generating a first image by reducing the camera image based on the taken camera image and a second image by cutting out a portion of the distant region from the camera image, inputting both the first image and the second image into a predetermined learning model to detect objects, and outputting a combination of the detection results based on the first image and the second image. [Effects of the Invention]

[0013] According to the present invention, it is possible to easily set an appropriate far area to achieve high-speed and high-precision object detection, regardless of the installation conditions of the camera. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an overview of object detection by a first conventional method. [Figure 2] FIG. 10 is a diagram illustrating an outline of object detection by a second conventional method. [Figure 3] FIG. 10 is a diagram illustrating an overview of object detection by a third conventional method. [Figure 4] 1 is a diagram illustrating an example of the configuration of an object detection system according to an embodiment of the present invention. [Figure 5] 10A and 10B are diagrams illustrating an example of setting a detection area and an object frame for a camera image. [Figure 6] FIG. 10 is a diagram illustrating an example of setting a far region for a camera image. [Figure 7] FIG. 10 is a diagram showing another example of setting a far region for a camera image. [Figure 8] FIG. 10 is a diagram showing another example of setting an object frame for a camera image. DETAILED DESCRIPTION OF THE INVENTION

[0015] An embodiment of the present invention will be described with reference to the drawings. FIG. 4 shows an example of the configuration of an object detection system according to an embodiment of the present invention. As shown in FIG. 4, the object detection system of this example includes an imaging device 110, an image processing device 120, a monitor terminal 130, and an operation terminal 140. These devices can be connected to each other via wire or wirelessly so that they can communicate with each other. Furthermore, any network such as the Internet may be interposed between these devices.

[0016] The imaging device 110 is a device such as a surveillance camera that captures images of a monitored area. In this example, a camera capable of outputting high-quality camera images such as 4K video is used as the imaging device 110. The imaging device 110 can be installed under installation conditions that correspond to the situation on-site. Therefore, the angle of view and tilt of the imaging device 110 are not particularly limited. However, for the sake of simplicity, it is assumed that the imaging device 110 in this example is installed in a substantially tilt-free position and captures camera images capturing the monitored area with a horizontal or nearly horizontal line of sight. The high-resolution camera images captured by the imaging device 110 are transmitted to the image processing device 120.

[0017] The image processing device 120 is, for example, a computer equipped with hardware resources such as a processor and memory, and is configured to read from the memory programs relating to the following functions of the present invention and execute them with the processor. The image processing device 120 has a function of performing object detection using a method similar to the third conventional method described above based on high-resolution camera images received from the imaging device 110 during system operation. The image processing device 120 also has a function of setting a far area. The far area is set before the system starts operating. Setting of the far area will be described in detail below.

[0018] Here, the learning model used by the image processing device 120 is for detecting people included in images, and has, for example, a yolo v3 network structure, and is trained using an input image of 640 x 360 pixels. Furthermore, a threshold representing the size (height) of a person in an image that can be reliably detected by the learning model is defined as Th, and is assumed to be Th = 50 pixels. In other words, people whose size in an image is 50 pixels or larger can be reliably detected, but detection accuracy decreases when the size is smaller than that. Note that the threshold Th = 50 pixels is just an example, and will vary depending on the structure of the learning model and the data provided during learning.

[0019] When setting the far area, the image processing device 120 receives from the user a detection area to be targeted for person detection in a high-resolution camera image captured in advance by the imaging device 110. Similarly, multiple object frames surrounding each of multiple people included in the camera image are received from the user. In FIG. 5, one detection area 210 and two object frames 221, 222 are set for a camera image 200 captured in advance. Note that detection areas may be set in two or more locations. Also, object frames may be set in three or more locations.

[0020] These settings are input by the user via operation terminal 140 and provided to image processing device 120. In this example, image processing device 120 is configured to display camera image 200 on operation terminal 140 and accept settings of the detection area and object frame through operations on that image. Note that the above description is an example, and the method for setting the detection area and object frame is not particularly limited.

[0021] After receiving the detection area and object frame settings, the image processing device 120 estimates the size of the person at each coordinate within the detection area by linear interpolation. As an example, in a 4K (3840 × 2150 pix) camera image, the Y coordinate of the top of the detection area is y_u = 400, the Y coordinate of the bottom is y_b = 2150, the Y coordinate of the first object frame is y1 = 500, the height h1 = 75 pix, and the Y coordinate of the second object frame is y2 = 1000, the height h1 = 450 pix. Furthermore, the height of each person is h = 170 cm. Furthermore, if the resize factor is defined as r, when converting a 4K (3840 × 2150 pix) camera image to a predetermined size (e.g., 640 × 360 pix), the resize factor r = 640 / 3850 ≒ 0.1667.

[0022] Here, the height H of a person at any Y coordinate can be expressed by the following formula: p1=h1 / h p2=h2 / h H=(Y×(p2-p1) / (y2-y1) +(p2-(p2-p1) / (y2-y1)×y1))×h×r

[0023] According to the above formula, when Y = 199.94, H = 50, which is the same value as the threshold value Th. Therefore, in an image reduced from 3840 x 2160 to 640 x 360, positions where Y = 200 or greater are the range in which people can be reliably detected. Therefore, if the rest of the image range, i.e., the image range from (0,0) to (3840,200), is set as the distant region, the maximum conditions can be met in all areas of the camera image 200.

[0024] In the example of FIG. 6, 14 far area frames 230 (7 horizontal × 2 vertical) each having a size of 640 × 360 are set for the image range of (0,0) to (3840,200). In other words, the system is set so that 14 cropped images are cropped from the camera image when it is in operation. Note that, as shown in FIG. 6, multiple far area frames 230 may be arranged so that the boundaries of adjacent frames overlap. This makes it possible to avoid a decrease in detection accuracy due to the person being cut off in the cropped image when a person is present on the boundary line between adjacent far area frames.

[0025] Here, the size of the far region frame may be other sizes as long as the value satisfies the above formula. For example, the image may be cropped to a size larger than the expected input size of the low-resolution learning model, such as cropping to a size of 1280 x 720 and resizing to 640 x 360, and then reduced to match the expected input size. Also, as shown in FIG. 7, the far region frame 230 may not be set outside the detection area 210 in the camera image 200. In FIG. 7, fewer than the 14 far region frames 230 in FIG. 6, i.e., seven far region frames 230, are set.

[0026] The image processing device 120 performs the above processing before system operation to set a far region for the camera image of the imaging device 110. During subsequent system operation, the image processing device 120 performs object detection on the camera image received from the imaging device 110 using a method similar to the third conventional method described above, according to the far region setting. That is, the image processing device 120 generates a reduced image obtained by reducing a high-resolution camera image and a cropped image obtained by cutting out a portion of the far region from the camera image. In the case of the setting shown in FIG. 6, 14 cropped images are generated, and in the case of the setting shown in FIG. 7, 14 cropped images are generated. The image processing device 120 inputs both the reduced image and the cropped image into a low-resolution learning model to perform person detection. The image processing device 120 then combines the detection result based on the reduced image with multiple detection results based on multiple cropped images, and outputs the combined image as a final output image. The final output image output from the image processing device 120 is transmitted to the monitor terminal 130 and displayed on the monitor terminal 130.

[0027] Here, the reduced image in the object detection described above may be reduced to the expected input size of the low-resolution learning model after removing areas other than the detection area. Furthermore, if the aspect ratio of the image after removing areas other than the detection area differs from that of the expected input size, the range to be removed may be expanded so that the aspect ratio matches. Alternatively, the image may be reduced while maintaining the aspect ratio after removal, and the missing portion may be padded.

[0028] As described above, the image processing device 120 of this example has the functions of: (1) setting a far region including an area in the image region of a camera image (200) previously captured by the imaging device 110 where the size of the object frame is equal to or smaller than a threshold value (Th) based on the positions and sizes of multiple object frames (221, 222) surrounding each of multiple people included in the camera image (200); (2) generating a reduced image by reducing the camera image and a cropped image by cutting out a portion of the far region from the camera image based on a camera image captured during system operation; (3) inputting both the reduced image and the cropped image into a low-resolution learning model to perform person detection; and (4) outputting a composite detection result based on the reduced image and the cropped image. This configuration makes it possible to easily set an appropriate far region to achieve high-speed, high-precision object detection regardless of the installation conditions of the imaging device 110.

[0029] Here, in the above description, a person detection system that detects a person from a camera image has been used as an example, but the present technology can be applied to any object detection system that detects various other objects. Furthermore, in the above description, it is assumed that the image capture device 110 is installed in a substantially tilted position, but the installation manner of the image capture device 110 is not limited to this, and for example, the image capture device 110 may be installed at an angle. In this case, by setting at least three object frames for the camera image, it is possible to appropriately set the far area.

[0030] Furthermore, in the above description, the object frame is set by the user operating the operation terminal 140. However, it is also possible to automate the setting of the object frame. Specifically, for example, as shown in FIG. 8, a plurality of provisional far area frames 240 are set to cover the entire image for a camera image previously captured by the imaging device 110, and the system is then trial-run. In the example of FIG. 8, a total of 35 provisional far area frames 240 are set, with 7 frames horizontally and 5 frames vertically. Trial-running the system with such settings increases the processing load on the image processing device 120 and takes a certain amount of time, but enables accurate detection of people included in the camera image. Therefore, it is possible to automatically set multiple object frames surrounding each person without user operation. Automating the setting of the object frame is particularly effective when the orientation of the imaging device 110 is changed during system operation.

[0031] Furthermore, although the imaging device 110 and the image processing device 120 are described as separate devices in the above description, these devices may be integrated. That is, the imaging device 110 may have not only a function for capturing camera images, but also a function for setting a distant region including a region in the image region of a camera image where the size of an object frame is equal to or smaller than a threshold based on the positions and sizes of multiple object frames surrounding each of multiple people included in the camera image captured before the system is put into operation, and a function for generating, based on a camera image captured during system operation, a reduced image by reducing the camera image and a cropped image by cutting out a portion of the distant region from the camera image, inputting both the reduced image and the cropped image into a low-resolution learning model to perform person detection, and outputting a combined detection result based on the reduced image and the detection result based on the cropped image.

[0032] Although the embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take on various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and modifications thereof are included in the scope and spirit of the invention described in this specification, etc., and are included in the invention described in the claims and their equivalents.

[0033] Furthermore, the present invention can be provided not only as devices such as those described above or as systems composed of these devices, but also as methods executed by these devices, programs for realizing the functions of these devices using a processor, and storage media for storing such programs in a computer-readable manner. [Industrial Applicability]

[0034] The present invention relates to an object detection system that detects an object from a camera image of a monitoring area. [Explanation of symbols]

[0035] 110: Imaging device, 120: Image processing device, 130: Monitor terminal, 140: Operation terminal

Claims

1. In an object detection system that detects objects from camera images of a monitoring area, Before starting operation, a process is executed to set a distant region including a region in an image region of a camera image where the size of the object frame is equal to or smaller than a threshold value, based on the positions and sizes of a plurality of object frames surrounding each of a plurality of objects included in a camera image previously captured; An object detection system characterized by the following: during operation, based on a captured camera image, a first image is generated by reducing the size of the camera image, and a second image is generated by cutting out a portion of the distant area from the camera image; both the first image and the second image are input into a predetermined learning model to detect objects; and the detection results based on the first image and the detection results based on the second image are synthesized and output.

2. 2. The object detection system according to claim 1, An object detection system, characterized in that the plurality of object frames are set by a user operation on the pre-captured camera image.

3. 2. The object detection system according to claim 1, An object detection system characterized in that the multiple object frames are set based on detection results obtained by inputting the previously captured camera images into the learning model.

4. In a camera that photographs a monitoring area to detect an object, a function of setting a distant area including an area in the image area of ​​a camera image where the size of the object frame is equal to or smaller than a threshold value based on the positions and sizes of a plurality of object frames surrounding each of a plurality of objects included in the camera image captured before the start of operation; A camera characterized by having the function of generating, based on a camera image taken during operation, a first image that is a reduced version of the camera image and a second image that is a cut-out portion of the distant area from the camera image, inputting both the first image and the second image into a predetermined learning model to detect objects, and combining and outputting the detection results based on the first image and the detection results based on the second image.

5. An object detection method for detecting an object from a camera image of a monitoring area, Before starting operation, a step of setting a distant area including an area in an image area of ​​a camera image where the size of the object frame is equal to or smaller than a threshold value based on the positions and sizes of a plurality of object frames surrounding each of a plurality of objects included in a camera image captured in advance; An object detection method characterized by comprising the steps of: generating, during operation, a first image obtained by reducing a captured camera image and a second image obtained by cutting out a portion of the distant area from the camera image; inputting both the first image and the second image into a predetermined learning model to detect an object; and combining and outputting the detection results based on the first image and the detection results based on the second image.

Images