OBJECT DETECTION DEVICE, OBJECT DETECTION METHOD, AND OBJECT DETECTION PROGRAM

Abstract

The present invention can achieve object detection with high precision while maintaining a fixed processing speed even in an environment with limited resources.　This object detection device includes: a rectangle extracting unit that extracts, from an input image, a plurality of rectangles that serve as candidates for applying object detection; a rectangle selecting unit that selects a fixed number of rectangles for applying object detection, from the rectangle candidates extracted by the rectangle extracting unit; and an object detecting unit that performs object detection on the rectangles selected by the rectangle selecting unit, and outputs, as an object detection result, metadata that includes at least the classes of objects included in the input image, the reliability, and bounding boxes.

Description

[Technical Field]

[0001] The disclosed technology relates to an object detection device, an object detection method, and an object detection program. [Background technology]

[0002] There is a conventional technology related to object detection devices. An object detection device is a device that estimates the class (person, car, etc.) and bounding box of an object contained in an input image, as well as its reliability. A bounding box is the coordinate information of a rectangle that surrounds an object.

[0003] In recent years, several object detection models using deep learning have been proposed. Deep learning-based object detection models include YOLO (You Only Look Once) and RetinaNet (see Non-Patent Documents 1 and 2), which infer bounding boxes and object classes simultaneously. Other proposed object detection models include Faster R-CNN, which performs object candidate region detection and class classification separately, and Faster R-CNN, an improved version of Faster R-CNN (see Non-Patent Documents 3 and 4). When deep learning-based object detection models first emerged, they required a large amount of computation and time for inference. However, improvements to the learning method and neural network structure have significantly improved inference speed and accuracy.

[0004] Additionally, several methods have been proposed for detecting objects from high-resolution images or videos by dividing the image. For example, a method has been proposed in which a group of images divided equally to match the input size of the object detection model and an entire image reduced in size are input to the object detection model (see Non-Patent Document 5). This technology scales the coordinate information of the obtained bounding box, then combines the detection results of each divided and reduced image, and outputs the final result. Another method has been proposed in which the distribution of objects is estimated using density estimation or cluster detection, and then the image is divided based on this information and an object detection model is applied (see Non-Patent Document 6 and Non-Patent Document 7). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] J. Redmon et al., "You Only Look Once: Unified, Real-Time Object Detection," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788. [Non-patent document 2] T. -Y. Lin et al., "Focal Loss for Dense Object Detection," 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2999-3007. [Non-patent document 3] R. Girshick et al., "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation," 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 580-587. [Non-patent document 4] S. Ren et al., "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 1 June 2017. [Non-patent document 5] H. Uzawa et al., “High-definition object detection technology based on AI inference scheme and its implementation”, IEICE Electronics Express, 2021, Volume 18, Issue 22, Pages 20210323. [Non-patent document 6] C. Li et al., "Density Map Guided Object Detection in Aerial Images," 2020 IEEE / CVF CVPRW, 2020, pp. 737-746. [Non-Patent Document 7] F. Yang et al., "Clustered Object Detection in Aerial Images," 2019 IEEE / CVF ICCV, 2019, pp. 8310-8319. Summary of the Invention [Problem to be solved by the invention]

[0006] As described above, conventional methods for detecting objects from high-definition video include methods for detecting objects without dividing the image, and methods for dividing the image and applying object detection to each divided image. Image division methods can be further divided into methods that divide the image equally and methods that divide the image adaptively.

[0007] The method of dividing an image evenly has the problem that the number of divisions becomes very large for high-definition images, resulting in large errors in the synthesis of the results. Furthermore, the adaptive division method has the potential to significantly reduce the number of divisions in some scenes, but for some images, the number of divisions may be the same as for equal division, which may result in the inability to process all divided images within the desired processing time, resulting in a decrease in the accuracy of the detection results. This problem is particularly pronounced in environments with limited computing resources, such as edge devices.

[0008] The disclosed technology has been made in consideration of the above points, and aims to provide an object detection device, an object detection method, and an object detection program that can achieve high-precision object detection while maintaining a constant processing speed even in an environment with limited resources. [Means for solving the problem]

[0009] An object detection device in a first aspect of the present disclosure includes a rectangle extraction unit that extracts a plurality of rectangles from an input image as candidates for applying object detection to the rectangles; a rectangle selection unit that selects a certain number of rectangles for applying object detection to the rectangle candidates extracted by the rectangle extraction unit; and an object detection unit that performs object detection on the rectangles selected by the rectangle selection unit and outputs metadata including at least the class, confidence, and bounding box of the object included in the input image as the object detection result.

[0010] An object detection method according to a second aspect of the present disclosure causes a computer to execute the following process: extracting from an input image a plurality of rectangles that are candidates for object detection; selecting a certain number of rectangles from the extracted candidate rectangles to which object detection is to be applied; performing object detection on the selected rectangles; and outputting metadata including at least the class, confidence, and bounding box of the object contained in the input image as the object detection result.

[0011] An object detection program in a third aspect of the present disclosure causes a computer to execute the following process: extracting multiple rectangles from an input image as candidates for applying object detection; selecting a certain number of rectangles from the extracted rectangle candidates to apply object detection; performing object detection on the selected rectangles; and outputting metadata including at least the class, confidence, and bounding box of the object contained in the input image as the object detection result. [Effects of the Invention]

[0012] According to the disclosed technology, highly accurate object detection can be achieved while maintaining a constant processing speed even in an environment with limited resources. [Brief explanation of the drawings]

[0013] [Figure 1A] FIG. 10 is a configuration diagram of a process for equally dividing an image and detecting an object. [Figure 1B] FIG. 1 is a configuration diagram of a method for adaptively dividing an image by estimating the distribution of objects. [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of the object detection device. [Figure 3] 1 is a block diagram showing the configuration of an object detection device according to a first embodiment. [Figure 4] FIG. 2 is a block diagram showing a hardware configuration of the object detection device. [Figure 5] 10 is a flowchart showing the flow of an object detection process performed by the object detection device. [Figure 6] 10 is a detailed flow diagram for rectangular selection processing when a method using detection results from past frames is applied. [Figure 7] FIG. 10 is a diagram showing an example in which an input image is divided equally into four sections and this cyclic method is applied. [Figure 8] FIG. 10 is a block diagram showing the configuration of an object detection device 200 according to a second embodiment. [Figure 9] 10 is a flowchart showing a case where thinning-out determination is performed at regular intervals. [Figure 10] 10 is a flowchart for detecting a decrease in the number of detected objects and making a thinning-out determination. [Figure 11] 10 is a flowchart showing a case where a thinning determination is made by combining a method of a fixed time and a method of the number of detections. [Figure 12] 10 is a flowchart of a process for predicting the movement of a rectangle. [Figure 13] This is an example of processing input frames using a pipelined processing flow. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0015] First, a prerequisite technology for the technology proposed in the embodiment of the present disclosure and an overview of the embodiment will be described.

[0016] As the object detection models mentioned in the conventional technology section have been improved and their detection accuracy has increased, there has been a growing movement to apply AI inference technology, including object detection, to industrial fields such as autonomous driving and IoT. AI inference technology can be broadly divided into cloud AI and edge AI, depending on whether inference is performed on the cloud or on the device.

[0017] Cloud AI is provided by services such as Google Cloud Platform (GCP), Amazon Web Services (AWS), and Microsoft Azure. Cloud AI performs inference processing such as object detection using large-scale computing resources on servers equipped with GPUs (Graphics Processing Units). Edge AI, on the other hand, performs inference processing on devices at the edge of the network, such as smartphones or drones. Compared to cloud AI, edge AI performs inference processing on devices such as smartphones or drones located at the edge of the network. Because of limited computing resources, such as memory size and processor performance, edge AI is not suitable for running large-scale AI inference models. However, it can minimize information exchange over the Internet and reduce communication costs, offering significant advantages in terms of security measures and cost reduction. Taking advantage of these characteristics, research and development is underway to apply edge AI to autonomous driving, crime prevention, and quality assurance and safety management in manufacturing. Object detection is widely used in these applications and forms the core of AI inference technology. For example, edge AI is used in applications where small cameras and processors are installed in surveillance cameras and drones to monitor and track people, vehicles, etc.

[0018] Traditionally, cameras installed on edge devices have not had very high resolution. However, as camera sensors have become smaller and more powerful, drones and surveillance cameras equipped with 4K cameras have become commonplace, and recently, smartphones and drones equipped with even higher-resolution 8K cameras have also appeared. Therefore, demand for devices capable of performing object detection from such high-resolution video is expected to increase in the future. However, many object detection models have a fixed input size and cannot process high-resolution images as is. For example, the input size of the YOLO v3 object detection model is generally around 500 to 1500 pixels. Some object detection models that use fully convolutional networks (FCNs) can handle variable input sizes. Therefore, even high-resolution images such as 8K can be input as is or with a low reduction ratio. However, as the resolution of input images increases, the intermediate features and the model itself become larger. This makes it impractical to perform object detection directly from high-resolution images, especially on edge devices with limited computational resources. Therefore, methods have been proposed to achieve object detection from high-resolution images and videos by dividing the image, as listed in Non-Patent Documents 5 to 7. Below, we will explain (1) the method of dividing the image evenly and (2) the method of dividing the image adaptively.

[0019] (1) Dividing the image equally (conventional method 1) FIG. 1A is a block diagram of a process for evenly dividing an image and performing object detection. This block diagram is, for example, the method described in Non-Patent Document 5. Conventional method 1 is configured with a segmentation processor, a general processor, and a synthesis processor. The segmentation processor divides the image evenly and performs object detection from each divided image. Meanwhile, the general processor reduces the entire image and applies object detection. Finally, the synthesis processor combines the results obtained by the segmentation processor with the results obtained by the general processor, scaled to the original image size, to output the final object detection result. With this method, when performing object detection on a 4K (3840 × 2160) image using YOLO v3 with an input image size of 608 × 608, the number of divisions is 28. On the other hand, for an 8K (7680 × 4320) image, the number of divisions is four times as large, at 112, resulting in a massive computational load in the segmentation processor. Furthermore, the number of division boundaries that require bounding box synthesis increases, resulting in a larger number of cut objects. As a result, errors accumulate in the synthesis processing unit, and the accuracy of the object detection that is finally output decreases.

[0020] (2) Dividing the image equally (conventional method 2) Figure 1B shows the configuration of a method for adaptively segmenting an image by estimating the distribution of objects. Conventional method 2 consists of two functional sections: a rectangle extraction section and an object detection section. First, the rectangle extraction section reduces the size of the input image and estimates the distribution of objects using methods such as density estimation and cluster detection. Based on the results, it then determines the regions (rectangles) to which object detection is applied according to the object distribution. The object detection section then extracts the rectangles from the input image and applies object detection to each of them. Because the rectangles are extracted according to the distribution of objects, the cutoff of objects that occurs when applying an equal segmentation method is unlikely to occur. However, the number of image segments can vary significantly depending on the distribution of objects. In situations where objects are concentrated in a certain part of the image, it may be possible to minimize the number of segments to one. However, in the worst case, the number of segments will be the same as with equal segmentation, resulting in no reduction in computational effort. When the number of image segments increases, object detection may not be completed within the desired processing time in environments with limited computational resources, such as edge AI execution environments.

[0021] Furthermore, the object detection method using equal division in conventional method 1 has the problems of an increase in the amount of calculations as the number of divisions increases, and a decrease in accuracy due to object cut-off. Of these, adaptive division mitigates the decrease in accuracy due to object cut-off, but does not necessarily reduce the amount of calculations. In other words, there is a problem in that it is difficult to reduce the number of rectangles to which object detection is applied to a certain number while suppressing the decrease in object detection accuracy and suppress the increase in calculations.

[0022] The method of this embodiment has been devised to solve the above-mentioned problems. The method of this embodiment calculates a priority score for multiple rectangles extracted by a rectangle selection unit based on information such as object density and past frames, and narrows down the rectangles to a certain number, thereby reducing the number of rectangles to which object detection is applied. This makes it possible to perform object detection within a specified processing time while suppressing a decrease in object detection accuracy, even in environments with limited computational resources, such as edge devices.

[0023] 2 is a block diagram showing the hardware configuration of the object detection device 100 according to the embodiment of the present disclosure.

[0024] 2, the object detection device 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0025] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 controls each of the above components and performs various arithmetic processing in accordance with the program stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores an object detection program.

[0026] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0027] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

[0028] The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may function as the input unit 15 by adopting a touch panel system.

[0029] The communication interface 17 is an interface for communicating with other devices such as terminals, etc. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0030] [First embodiment] Next, each functional configuration of the object detection device 100 according to the first embodiment will be described. Fig. 3 is a block diagram showing the configuration of the object detection device according to this embodiment. Each functional configuration is realized by the CPU 11 reading out an object detection program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

[0031] Fig. 3 shows a configuration diagram of object detection device 100 that realizes the first embodiment. As shown in Fig. 3, object detection device 100 is configured to include a rectangle extraction unit 110, a rectangle selection unit 112, and an object detection unit 114. Object detection device 100 receives a series of input images as video input, and executes processing in each unit for each input image.

[0032] The first embodiment is similar to conventional method 2 in that it adaptively divides an image according to the distribution of objects and performs object detection. However, it differs in that it newly includes a rectangle selection unit that selects rectangles to which object detection is applied.

[0033] The rectangle extraction unit 110 extracts multiple candidate rectangles (hereinafter simply referred to as candidate rectangles) by estimating the distribution of objects. The input image is reduced to a certain size, and a deep learning model such as object detection or cluster detection is used to estimate the area where objects exist as a distribution of the input image. When density estimation is used to estimate the distribution, an area where a density distribution equal to or greater than a predetermined value is cut out and extracted as a candidate rectangle for object detection, and coordinate information is obtained. When cluster detection is used, a deep learning model is used to estimate the coordinates and reliability of clusters where objects are densely concentrated, and clusters with reliability equal to or greater than a certain value are extracted as candidate rectangles.

[0034] The rectangle selection unit 112 selects rectangles to which object detection is to be applied from among the candidate rectangles extracted by the rectangle extraction unit 110, using the density estimation result and rectangles selected in past frames. For the selection, the density score s density and the multiplicity score s iou The priority is calculated from the above, and the rectangle is selected according to the ranking of the priority. The selection method will be described in detail later in the explanation of the flow.

[0035] Finally, the object detection unit 114 applies an object detection model to each of the rectangles selected by the rectangle selection unit 112, and outputs a final object detection result. Any object detection model can be selected here. The object detection result is output as metadata including at least the class, confidence, and bounding box of the object included in the input image.

[0036] In this embodiment, the rectangle selection method is a method of selecting rectangles to which object detection is applied using the results of density estimation and rectangles selected in past frames, but this is not limited to this, and methods using detection results from past frames, methods using image differences, methods of cyclically selecting rectangles, and methods combining these may also be used.

[0037] Next, a description will be given of the operation of object detection device 100. Fig. 4 is a flowchart showing the flow of object detection processing by object detection device 100. The object detection processing is performed by CPU 11 reading out an object detection program from ROM 12 or storage 14, expanding it into RAM 13, and executing it.

[0038] In step S100, the CPU 11 functions as the rectangle extraction unit 110 to extract a plurality of candidate rectangles by estimating the distribution of objects.

[0039] In step S102, CPU 11 functions as rectangle selection unit 112 to select, from among the candidate rectangles obtained by rectangle extraction unit 110, rectangles to which object detection is to be applied using the result of density estimation and rectangles selected in previous frames.

[0040] In step S104, the CPU 11 functions as the object detection unit 114 to apply an object detection model to each of the rectangles selected by the rectangle selection unit 112, and outputs a final object detection result.

[0041] Next, we will explain the detailed flow of the rectangle selection process by the rectangle selection unit in step S102. There are several selection methods: a method that uses the results of density estimation, a method that uses the detection results of past frames, a method that selects rectangles based on image differences, and a method that divides the input image into multiple partitions and cyclically selects partitions to select rectangles included in those partitions.

[0042] Of these, the method using the density estimation result and the method using the detection result of a past frame will be explained using a flow. With reference to Figure 5, a detailed flow will be explained when the method using the density estimation result is applied to the rectangle selection process of step S102. Note that the process is performed for each extracted rectangle.

[0043] In step S200, first, the density values ​​within the rectangle extracted in the current frame are summed to obtain a density score s density In density estimation, the density estimate d is calculated for a pixel at position (x,y) on the input image. x,y is assigned. The extracted rectangle R i The set of pixel coordinates (x, y) contained in (i=1,…,N) is R i (In the formula, it is also written as Ri when it is a subscript), and the density score of a rectangle is s density =Σ (x,y)∈Ri d x,y is given by

[0044] Next, in step S202, the rectangle selected in the previous frame and the rectangle R extracted from the current frame are compared. i Calculate the overlap (IoU: Intersection over Union) of iou Given two rectangles with areas a1 and a2, their IoU is calculated as the area of ​​the overlapping part a inter Using a inter / ((a1+a2-a inter ) can be calculated. This value is calculated for each pair with the rectangle selected in the previous frame, and the maximum value is calculated for the rectangle R i The multiplicity score s of iou Let's say.

[0045] In step S204, a priority score s is calculated from the obtained density score and overlap score. priority Here, we preferentially select rectangles extracted from areas that have not been detected before and rectangles with high object density, so we calculate s priority =-λs iou +s density and priority =1 / s iou +λs density where λ is a parameter used to calculate the weighted sum, and s iou and s density Either of these coefficients may be used, or they may be multiplied by the respective coefficients such as λ1 and λ2.

[0046] In step S206, a ranking is created in descending order of the priority score.

[0047] In step S208, it is determined whether the rectangle is among the top rankings. If it is among the top rankings, the rectangle is extracted in step S210; if it is not among the top rankings, the process ends. Rectangles are selected from the top, so that the number of rectangles is a predetermined number taking into account the application and the hardware configuration of the device. This rectangle selection method applies object detection to areas where object detection has not been applied before, in addition to areas where a high number of dense object detections are expected. In this way, the rectangle selection unit 112 can use a method of selecting rectangles based on the degree of overlap between the distribution estimation result obtained from the rectangle extraction unit 110 and rectangles selected in previous input images.

[0048] Next, a method using the detection results of past frames will be described. A detailed flow for applying the method using the detection results of past frames to the rectangular selection process in step S102 will be described with reference to Fig. 6. Note that only step S200 differs from the flow in Fig. 5, and so only this step will be described as step S300.

[0049] In the method using the detection results of past frames, the number of detected objects is counted and a priority score s priority Calculate the priority score s priority In the calculation of , not only the rectangle selected in the past frame but also the coordinates of the detected object are recorded. Also, in the calculation, in step S300, the density score s density The calculation of the object number score s is calculated by counting the number of objects detected in the rectangle from the previous frame. obj_num In this case, for a while after the video input starts, the priority score is calculated using s obj_num Set s to 0, perform rectangular selection and object detection, and grasp the object coordinates in the entire image. obj_num The period in which s is set to 0 may be set to any number of frames in advance, or s obj_num This process may be repeated any number of times until there are no rectangles for which σ is 0, or until the number is equal to or less than a certain number. Furthermore, for rectangles to which object detection has been applied, the coordinate information of the object detection result for the corresponding area on the input frame is updated. This method is suitable for applications that place importance on the number of detected objects. In this way, the rectangle selection unit 112 can use a method for selecting rectangles based on the object detection results obtained from past input images.

[0050] In the image difference method, the priority is determined based on the difference between the previous frame and the current frame. First, each frame image is converted from a color image in RGB format to grayscale. Then, the difference is calculated on a pixel-by-pixel basis, and a difference image is generated with the absolute value of the difference as the pixel value. The difference image is cut out based on the coordinates of the rectangle obtained from the current frame, and the sum of the pixel values ​​is calculated to determine the priority score s priority In other words, object detection is applied preferentially to rectangles with large image differences caused by object movement. This method is suitable for applications that detect moving objects, such as moving cars and walking people. In this way, the rectangle selection unit 112 can use a method of selecting rectangles based on image differences from past input images.

[0051] In the method of cyclic rectangle selection, for example, an image is divided into N sections, and rectangles contained in a certain section are selected preferentially. The section to be set with high priority is set cyclically, with section 1 being given priority at time t, section N being given priority at time t+N-1, and so on, returning to section 1 at time t+N. The method of determining sections in this method is arbitrary, and the sections may be set evenly, or unevenly depending on the scene. For example, Figure 7 shows an example in which this cyclic method is applied to an input image divided evenly into four sections.

[0052] In this example, the section that was in the upper left corner at time t and from which rectangles are selected with priority moves through each section in turn until time t+3, returning to the original upper left corner at time t+4. This method is suitable for detecting the entire image evenly, and is useful in situations where objects do not move very vigorously and are distributed throughout the image. In this way, the rectangle selection unit 112 can use a technique of dividing the input image into multiple sections, selecting sections in a cyclical manner, and selecting rectangles contained in those sections.

[0053] The above rectangle selection methods are not mutually exclusive and may be used in combination in some cases. For example, the priority score s priority The image difference value is calculated as the difference score s diff However, the combination of rectangular selection methods is not limited to this.

[0054] By processing images in this way, the problem of an increase in the number of divisions and the amount of calculations can be solved by always narrowing down the application of object detection to only a fixed number of rectangles in the rectangle selection unit 112. With this effect, it is possible to achieve highly accurate object detection while maintaining a constant processing speed, even in an environment with limited resources such as an edge terminal.

[0055] As described above, the object detection device 100 of this embodiment can achieve highly accurate object detection while maintaining a constant processing speed, even in an environment with limited resources.

[0056] [Second embodiment] FIG. 8 shows an example configuration of an object detection device 200 according to the second embodiment. In addition to the three processing units shown in the first embodiment, the object detection device 200 according to the second embodiment newly introduces a thinning determination unit 210 that determines whether to perform rectangle extraction and rectangle selection, i.e., whether to thin out the processing of the rectangle extraction unit 110 and the rectangle selection unit 112. Thinning out means omitting the processing for obtaining rectangles in the rectangle extraction unit 110 and the rectangle selection unit 112. The thinning determination unit 210 determines, using a predetermined method, whether to execute the processing in the rectangle extraction unit 110 and the rectangle selection unit 112. The thinning determination unit 210 then thins out the processing by applying the rectangles obtained by executing the processing in the rectangle extraction unit 110 and the rectangle selection unit 112 on a previously input frame to the current frame.

[0057] FIG. 9 shows a flowchart for performing a thinning determination after a fixed time interval. In step S400, it is determined whether a fixed time has elapsed since the previous rectangle selection. If the fixed time has elapsed, the process proceeds to step S402. If the fixed time has not elapsed, the process proceeds to step S404. In this thinning determination, the interval for extracting and selecting rectangles is set in advance as a hyperparameter according to the situation in which object detection is to be applied. If the fixed time has elapsed, rectangles are extracted and selected once (step S402). Then, the rectangle extraction unit 110 and the rectangle selection unit 112 are notified to thin out the processing by acquiring previously selected rectangles and performing object detection until the fixed time interval specified as the interval has elapsed (step S404). In this case, the processing and output of the rectangle extraction unit 110 and the rectangle selection unit 112 are temporarily stopped and thinned out, and the previously selected rectangles are used in the processing of the object detection unit 114. This thinning process eliminates the need to allocate computational resources to the extraction and selection of rectangles, including distribution estimation, and therefore allows those computational resources to be used for processing by the object detection unit 114, making it possible to apply object detection to a larger number of rectangles and expected to improve detection accuracy. In this way, the thinning determination unit 210 can use a method of realizing thinning by not performing processing by the rectangle extraction unit and the rectangle selection unit for a predetermined fixed period of time.

[0058] In the above example, the thinning determination is performed at regular time intervals. However, as shown in the flowchart of FIG. 10, the thinning determination may also be performed by detecting a decrease in the number of detected objects (step S500). In this method, the number of detected objects in a frame in which rectangles were extracted and selected is used as a reference, and if the number of detected objects in subsequent frames decreases by more than a certain amount, rectangles are extracted and selected again in the next frame. For this reason, a threshold value for the rate of decrease in the number of detected objects is set as a hyperparameter. In this way, the thinning determination unit 210 can use a method of implementing thinning processing by thinning out the processing of the rectangle extraction unit 110 and the rectangle selection unit 112 until the number of detected objects in a given frame falls below a certain rate, compared with the number of detected objects in the frame in which rectangles were extracted and selected.

[0059] The above methods may also be combined to perform the thinning decision (steps S400 and S500). FIG. 11 is a flowchart showing a case where a thinning decision is made by combining a fixed time and the number of detection methods. For example, a combination method may be used in which a long interval is set to forcibly execute rectangle extraction and selection, and rectangle extraction and selection is performed if the number of object detections decreases within that interval. In any of the above methods, for frames in which rectangle extraction and selection are not performed, the rectangle selected in the previous frame may be used as is, or the movement of the rectangle may be predicted and the coordinates of the rectangle to be cut out from the frame may be moved. Whether to perform prediction may be determined appropriately depending on the decrease in the number of detections.

[0060] A flowchart of the process for predicting rectangle movement is shown in Figure 12. In step S600, it is determined whether or not rectangle movement is to be predicted. If it is to be predicted, the process proceeds to step S602; if it is not to be predicted, the process proceeds to step S404. When predicting rectangle movement, rectangles are extracted and selected from consecutive frames over a certain period of time, and the rectangle movement is predicted based on the results (step S602). The prediction method may be linear interpolation, or a more accurate algorithm such as SORT may be used.

[0061] Each determination unit in the flowchart and rectangle movement prediction are performed by thinning-out determination unit 210 in the configuration diagram shown in Fig. 8, while acquisition of previous rectangles and rectangle extraction and selection processing are performed by rectangle extraction unit 110 and rectangle selection unit 112. As mentioned above, by allocating the surplus computational resources made available by thinning out rectangle extraction and selection to object detection, object detection can be applied to a larger number of rectangles, which is expected to improve object detection accuracy.

[0062] [Third embodiment] In the third embodiment, the object detection process shown in the first embodiment is pipelined to achieve efficient object detection. Specifically, object detection is performed from the frame at time t+1 using rectangles extracted and selected from the frame at time t. The flow of this process is shown in FIG. 12. This embodiment requires a device that can process inference of a deep learning model and other processes in parallel. Pipelining this process makes it possible to hide the waiting time due to rectangle selection, and object detection can be applied to even more rectangles than in the first and second embodiments, leading to improved detection accuracy and a wider range of application of this embodiment.

[0063] [Fourth embodiment] The fourth embodiment is a combination of the second and third embodiments. Specifically, object detection processing is performed while thinning out the processing of the rectangle extraction unit and rectangle selection unit at regular time intervals or based on the rate of decrease in the number of object detections. When object detection is required from consecutive frames, a pipelined processing flow is used to efficiently process the input frames. An example of such processing is shown in FIG. 13. In the section (a) where rectangle extraction and selection are performed in each frame, a pipelined processing flow is adopted. In the section (b) where rectangle extraction and selection are thinned out, rectangle movement prediction is performed. Rectangle extraction and selection are performed continuously from time t to time t+2, and rectangle movement prediction is performed from time t+3 to time t+5, and rectangle extraction and selection are thinned out. At this time, from time t to time t+2, processing is pipelined to efficiently extract rectangles, select them, and detect objects. This allows processing that takes advantage of the advantages shown in the second and third embodiments. In other words, by appropriately thinning out the extraction and selection of rectangles, the computational resources necessary to perform object detection from a larger number of rectangles are secured, while in situations where rectangle extraction and selection are required in consecutive frames, efficiency is improved through pipelining, reducing hardware waiting time and making it possible to apply object detection processing to a larger number of rectangles, leading to improved detection accuracy and expanded applications.

[0064] In this embodiment, the method for thinning out the rectangle extraction and selection processes uses one of the methods described in the second embodiment. Furthermore, the number of frames for which rectangle extraction and selection are performed consecutively can be set arbitrarily. If the conditions for the thinning process are no longer satisfied, rectangle extraction and selection may be performed consecutively again for any number of frames. In this way, the object detection device can have a pipelined processing mechanism such that the rectangles obtained by processing in the rectangle extraction unit 110 and the rectangle selection unit 112 for a frame input at time t-1 are applied to a frame input at time t, and processing in the object detection unit 114 is performed. Furthermore, the object detection device can also perform processing by combining the thinning determination unit 210 thinning out the processing in the rectangle extraction unit 110 and the rectangle selection unit 112 with a method for pipelined processing of each processing unit.

[0065] In the above embodiments, the object detection process executed by the CPU after reading the software (program) may be executed by various processors other than the CPU. Examples of such processors include dedicated electrical circuits, such as programmable logic devices (PLDs) (GPUs) whose circuit configuration can be changed after manufacture, such as field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs) that have circuit configurations specifically designed to execute specific processes. The object detection process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). The hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0066] In addition, in each of the above embodiments, the object detection program is described as being pre-stored (installed) in the storage 14, but the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0067] The following additional notes are provided regarding the above-described embodiments.

[0068] (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: Extract multiple rectangles from the input image that are candidates for object detection, From the extracted rectangle candidates, a certain number of rectangles are selected to apply object detection. Performing object detection on the selected rectangle, and outputting metadata including at least the class, confidence, and bounding box of the object included in the input image as the object detection result. The object detection device is configured as follows.

[0069] (Additional note 2) A non-transitory storage medium storing a program executable by a computer to perform an object detection process, Extract multiple rectangles from the input image that are candidates for object detection, From the extracted rectangle candidates, a certain number of rectangles are selected to apply object detection. Performing object detection on the selected rectangle, and outputting metadata including at least the class, confidence, and bounding box of the object included in the input image as the object detection result. Not a one-time memory medium.

Claims

1. a rectangle extraction unit that extracts a plurality of rectangles from an input image as candidates for object detection; a rectangle selection unit that selects a certain number of rectangles to which object detection is to be applied from among the rectangle candidates extracted by the rectangle extraction unit; an object detection unit that performs object detection on the rectangle selected by the rectangle selection unit and outputs metadata including at least the class, confidence, and bounding box of the object included in the input image as an object detection result; a thinning-out determination unit that determines whether or not to execute the processing in the rectangle extraction unit and the rectangle selection unit using a predetermined method, and thins out the processing by applying a rectangle obtained by executing the processing in the rectangle extraction unit and the rectangle selection unit on a previously input frame to a current frame; An object detection device comprising:

2. 2. The object detection device of claim 1, wherein the rectangle selection unit selects rectangles using one of the following methods, or a combination of these methods: a method of selecting rectangles based on the degree of overlap between the distribution estimation result obtained from the rectangle extraction unit and rectangles selected in the past input image; a method of selecting rectangles based on the object detection result obtained from the past input image; a method of selecting rectangles based on the image difference from the past input image; and a method of dividing the input image into a plurality of sections, selecting sections in a cyclical manner, and selecting rectangles included in the sections.

3. 2. The object detection device of claim 1, wherein the thinning determination unit thins out the processing of the rectangle extraction unit and the rectangle selection unit using either or both of a method of realizing thinning processing by not performing processing of the rectangle extraction unit and the rectangle selection unit for a predetermined fixed period of time, and a method of realizing thinning processing by thinning out the processing of the rectangle extraction unit and the rectangle selection unit until the number of object detections in a specified frame falls below a certain percentage compared to the number of objects detected in frames in which rectangles were extracted and selected.

4. The object detection device of claim 1 or claim 2, having a pipelined processing mechanism that applies the rectangles obtained by processing in the rectangle extraction unit and the rectangle selection unit for a frame input at time t-1 to a frame input at time t and executes processing in the object detection unit.

5. The object detection device according to claim 1 , wherein the thinning-out determination unit thins out the processing of the rectangle extraction unit and the rectangle selection unit, and the thinning-out determination unit performs processing in a pipelined manner.

6. extracting a plurality of rectangles from the input image as candidates for applying object detection, and selecting a certain number of rectangles to which object detection is to be applied from among the extracted rectangle candidates, determining whether or not to execute processing in a rectangle extraction and rectangle selection unit using a predetermined method, applying the rectangles obtained by executing the rectangle extraction and rectangle selection processes on a previously input frame to the current frame and thinning out the processing; Performing object detection on the selected rectangle, and outputting metadata including at least the class, confidence, and bounding box of the object included in the input image as the object detection result. An object detection method that causes a computer to perform processing.

7. extracting a plurality of rectangles from the input image as candidates for applying object detection, and selecting a certain number of rectangles to which object detection is to be applied from among the extracted rectangle candidates, determining whether or not to execute processing in a rectangle extraction and rectangle selection unit using a predetermined method, applying the rectangles obtained by executing the rectangle extraction and rectangle selection processes on a previously input frame to the current frame and thinning out the processing; Performing object detection on the selected rectangle, and outputting metadata including at least the class, confidence, and bounding box of the object included in the input image as the object detection result. An object detection program that causes a computer to perform processing.

Images