Object detection device and method

Abstract

This object detection device comprises: an overall processing unit (3) that performs object detection processing on an input image; an allocation processing unit (5) that pre-determines, in a cycle of N frames, a divided image for each frame to serve as a target of object detection among a plurality of divided images and reserves the determined divided image; a selection processing unit (2) that selects and outputs, for each frame, some divided images to serve as a target of object detection among the plurality of divided images on the basis of the reservation results of the allocation processing unit (5); a division processing unit (4) that performs object detection processing on each of the divided images output from the selection processing unit (2); and a synthesis processing unit (8) that synthesizes the object detection results from the overall processing unit (3) and the object detection results from the division processing unit (4).

Description

[Technical field]

[0001] The present invention relates to an object detection device and method for detecting an object from within an input image. [Background technology]

[0002] An object detection device is a device that detects metadata from an input image, consisting of the position of an object contained in the image (the position of the rectangular frame surrounding the object), attributes (type of object, such as person, car, etc.), and reliability.

[0003] In recent years, methods such as YOLO (You Only Look Once) and SSD (Single Shot multibox Detector) have been proposed to detect metadata based on deep learning results, and their use in AI (Artificial Intelligence) image processing in surveillance cameras and edge computing has been considered (see Non-Patent Documents 1 and 2). These object detection methods based on deep learning have restrictions on the input image size. For example, in YOLOv3 (Non-Patent Document 1), the input image size is either 320 pixels wide x 320 pixels high, 416 pixels wide x 416 pixels high, or 608 pixels wide x 608 pixels high.

[0004] In order to perform object detection based on deep learning for high-definition images such as full HD (Full High Definition) and 4K under the above-mentioned constraints on input image size, conventionally, the input image is reduced to the constrained image size and object detection is performed. This reduction also reduces the characteristic parts of the object, making it difficult to detect small objects included in the input image. A method of dividing the input image and performing object detection for each divided image has also been disclosed (see Non-Patent Document 3). However, for large objects that straddle divided images, the characteristic parts are also divided, making it difficult to detect large objects.

[0005] In order to make it possible to detect both large and small objects in a high-resolution image, a method can be considered in which both a whole process in which the input image is reduced to a constrained image size and object detection is performed on the whole image, and a division process in which the input image is divided and object detection is performed for each divided image, and the result of the whole process and the result of the division process are combined to obtain the final object detection result. This object detection method will be explained using Fig. 17.

[0006] In the overall process, image scaling is performed to reduce the input image to an image size that can be handled by object detection processing based on the deep learning results, and then detection processing is performed to detect objects contained in the reduced image. The rectangular frame (hereinafter referred to as BB) indicating the position of the detected object contains at least information on the center coordinates (X, Y) of the BB, the height (H) of the BB, and the width (W) of the BB. Since this information is for the reduced image, scaling of the BB (metadata scaling) is performed to map it to the original image before reduction. From the group of detected objects after scaling, objects with a reliability equal to or higher than a preset reliability threshold are selected and proceed to the subsequent synthesis process.

[0007] On the other hand, in the segmentation process, after the input image is segmented, a detection process is performed for each segmented image to detect objects contained in each segmented image. Since the BB indicating the position of the detected object is for the segmented image, metadata adjustment is performed to map it to the original image before segmentation. After the adjustment, as in the overall process, only objects whose reliability is equal to or greater than the reliability threshold are selected and proceed to the subsequent synthesis process.

[0008] In the synthesis process, objects not detected in the overall process are interpolated with objects detected in the segmentation process. In selecting objects to be interpolated, a match determination is made between the objects detected in the overall process and the objects detected in the segmentation process, and objects that do not match are output as objects to be interpolated. In the synthesis process with objects detected in the interpolation process, the output result of the metadata selection process for each segmented image and the output result of the overall process are combined and output as the final object detection result.

[0009] The method described in Figure 17 makes it possible to simultaneously detect both large and small objects even in high-resolution images that exceed the input image constraint size in object detection methods based on deep learning. However, there is a problem that the number of image divisions in the division process increases in ultra-high resolution images such as 4K (3840 pixels x 2160 pixels), resulting in a huge amount of calculation. For example, in the case of YOLOv3, even in YOLO with the largest input image constraint size, the maximum number of divided images is 18, resulting in a huge amount of calculation.

[0010] In order to reduce the amount of calculations, a method of dividing an image frame into an image frame for performing object detection and an image frame for tracking an object (Non-Patent Document 4) can be considered. In the frame for tracking an object, the position of the BB obtained by the frame for performing object detection is corrected based on the motion vector value. By applying the method disclosed in Non-Patent Document 4, the frames for performing object detection are thinned out, so that it is possible to reduce the amount of calculations compared to the case of performing object detection for each frame.

[0011] However, for object groups whose reliability is near a reliability threshold, adjacent object groups, and partially hidden object groups, the object detection results themselves are unstable, which causes a problem that frames in which an object could be detected are lost due to frame thinning, making it difficult to detect the object. [Prior art documents] [Non-patent literature]

[0012] [Non-Patent Document 1] Joseph Redmon et.al, “YOLOv3: An Incremental Improvement”, 2018,<https: / / arxiv.org / abs / 1804.02767> [Non-Patent Document 2] Wei Liu et.al, “SSD:Single Shot MultiBox Detector”, 2016,<https: / / arxiv.org / pdf / 1512.02325.pdf> [Non-Patent Document 3] Vit Ruzicka et.al,“Fast and accurate object detection in high resolution 4K and 8K video using GPUs”,2018 IEEE High Performance extreme Computing Conference (HPEC) [Non-Patent Document 4] Takayuki Ujiie et al., “Reducing the Load of CNN Object Detection Using Motion Vectors in Video Codecs”, IPSJ SIG Technical Report, Vol. 2018-CVIM-210, No. 4, 2018. Summary of the Invention [Problem to be solved by the invention]

[0013] The present invention has been made to solve the above-mentioned problems, and has an object to reduce the amount of calculations while suppressing the occurrence of missed object detection. [Means for solving the problem]

[0014] The object detection device of the present invention includes a division unit configured to divide an input image and output a plurality of divided images; an overall processing unit configured to perform object detection processing on a reduced image obtained by scaling the entire input image for each frame; an allocation processing unit configured to determine in advance, at a period of N frames (N is an integer equal to or greater than the total number of divided images), a divided image for each frame that is subject to object detection from among the plurality of divided images and to reserve the determined divided images; a first selection processing unit configured to select and output, for each frame, a portion of the divided images that are subject to object detection from among the plurality of divided images based on a result of the reservation by the allocation processing unit; and a division processing unit configured to perform object detection processing on each of the divided images output from the first selection processing unit.a storage unit configured to store the object detection results by the segmentation processing unit; and a second selection processing unit configured to select and output, for each frame, a past object detection result stored in the storage unit for a segmented image for which object detection is not performed, based on a result of the reservation by the allocation processing unit; The object detection result by the overall processing unit and the object detection result by the segmentation processing unit The object detection result output from the second selection processing unit; and a synthesis processing unit configured to synthesize the above-mentioned images for each frame to obtain a final object detection result.

[0015] In addition, in one configuration example of the object detection device of the present invention, the allocation processing unit is characterized in that it comprises an estimated fluctuation calculation unit configured to calculate an estimate of the number of detected objects for each divided image for each frame based on the object detection result by the overall processing unit, calculate an estimate of the fluctuation between frames of the number of detected objects for each divided image based on the absolute difference between the frames of the estimates, and calculate an average value of the estimated value of the fluctuation in the number of detected objects within the period for each divided image, and a divided image allocation processing unit configured to reserve, for the first selection processing unit, the divided images for each frame that will be the target of object detection in the next period based on the average value of the estimated value of the fluctuation in the number of detected objects for each divided image so that the number of times object detection is performed is proportionally distributed to each divided image. Furthermore, in one configuration example of the object detection device of the present invention, the divided image allocation processing unit determines that all of the divided images will be selected as targets for object detection at least M (M is an integer greater than or equal to 1) times during the next period, and then determines, for the remaining allocable frames, the divided images for each frame to be targets for object detection in the next period, based on the average value of the estimated fluctuation in the number of detected objects for each of the divided images, so that the number of times object detection is performed is proportionally distributed to each divided image. Furthermore, in one configuration example of the object detection device of the present invention, the estimated fluctuation calculation unit is characterized in that it calculates an estimate of the number of detected objects for each divided image by tallying up, for each divided image in which the objects are located, the number of detected objects among the group of objects detected by the overall processing unit, whose reliability calculated by the overall processing unit is less than a threshold value and greater than or equal to a threshold value / α (α is a real number greater than or equal to 2).

[0016] AlsoOne example configuration of the object detection device of the present invention further includes a scene change detection processing unit configured to determine whether the input image has changed based on the latest object detection result by the segmentation processing unit and past object detection results, and the allocation processing unit is characterized in that when it is determined that the input image has changed, it cancels the reservation and returns to an initial state.

[0017] The object detection method of the present invention includes a first step of dividing an input image and outputting a plurality of divided images; a second step of performing object detection processing on a reduced image obtained by scaling the entire input image for each frame; a third step of determining in advance, at a period of N frames (N is an integer equal to or greater than the total number of divided images), a divided image for each frame that is to be subject to object detection from among the plurality of divided images and reserving the determined divided images; a fourth step of selecting and outputting, for each frame, a portion of the divided images that are to be subject to object detection from among the plurality of divided images based on a result of the reservation in the third step; and a fifth step of performing object detection processing on each of the divided images output in the fourth step. a sixth step of storing the object detection results from the fifth step; and a seventh step of selecting and outputting the past object detection results stored in the sixth step for each frame for a divided image in which object detection is not performed based on the result of the reservation made in the third step; The object detection result by the second step and the object detection result by the fifth step The object detection result output by the seventh step; The final object detection result is obtained by synthesizing the 8 The present invention is characterized by including the steps of: Effect of the Invention

[0018] According to the present invention, by providing an allocation processing unit that determines in advance at an N-frame cycle those divided images for each frame that are to be subject to object detection from among the multiple divided images and reserves the determined divided images, and a first selection processing unit that selects and outputs, for each frame, some of the divided images from the multiple divided images that are to be subject to object detection based on the results of the reservation by the allocation processing unit, it is possible to reduce the amount of calculations while suppressing the occurrence of missed object detection. [Brief description of the drawings]

[0019] [Figure 1]FIG. 1 is a block diagram showing the configuration of an object detection device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram for explaining a divided image processed by the object detection device according to the first embodiment of the present invention. [Diagram 3] FIG. 3 is a flowchart illustrating a calculation process of an estimate of the number of detected objects in a divided image in the first embodiment of the present invention. [Figure 4] FIG. 4 is a diagram illustrating an example of an allocation process performed by the allocation processing unit according to the first embodiment of the present invention. [Diagram 5] FIG. 5 is a flowchart illustrating the operation of the divided image allocation processing unit according to the first embodiment of the present invention. [Figure 6] FIG. 6 is a flowchart illustrating reservation processing by the divided image allocation processing unit according to the first embodiment of the present invention. [Figure 7] FIG. 7 is a diagram for explaining a specific example of reservation processing by the divided image allocation processing unit according to the first embodiment of the present invention. [Figure 8] FIG. 8 is a flowchart illustrating the flow of object detection processing by the object detection device according to the first embodiment of the present invention. [Figure 9] FIG. 9 is a flowchart illustrating details of the overall processing by the overall processing unit according to the first embodiment of the present invention. [Figure 10] FIG. 10 is a flowchart illustrating details of the allocation process by the allocation processor according to the first embodiment of the present invention. [Figure 11] FIG. 11 is a flowchart illustrating details of the object detection process by the segmentation processing unit according to the first embodiment of the present invention. [Figure 12] FIG. 12 is a flowchart illustrating details of the synthesis process performed by the synthesis processing unit according to the first embodiment of the present invention. [Figure 13] FIG. 13 is a flowchart illustrating details of the metadata selection process by the synthesis processing unit according to the first embodiment of the present invention. [Figure 14] FIG. 14 is a block diagram showing the configuration of an object detection device according to the second embodiment of the present invention. [Figure 15] FIG. 15 is a flowchart illustrating the operation of the scene change detection processing unit according to the second embodiment of the present invention. [Figure 16] FIG. 16 is a block diagram showing an example of the configuration of a computer that realizes the object detection device according to the first and second embodiments of the present invention. [Figure 17] FIG. 17 is a diagram illustrating a conventional object detection method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] [First Example] Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an object detection device according to a first embodiment of the present invention. The object detection device executes object detection processing on an input image. The object detection device then outputs an object detection result from the object detection processing as metadata. The object detection device outputs metadata including at least position information of an object included in the input image, attribute information of the object, and a reliability.

[0021] The object detection device includes a division unit 1, a selection processing unit 2, an overall processing unit 3, a division processing unit 4, an allocation processing unit 5, a storage unit 6, a selection processing unit 7, and a synthesis processing unit 8.

[0022] The overall processing unit 3 includes an image scaling processing unit 30 , an object detection processing unit 31 , a metadata scaling processing unit 32 , and a reliability filter processing unit 33 . The division processing unit 4 includes an image scaling processing unit 40 , an object detection processing unit 41 , a metadata adjustment processing unit 42 , and a reliability filter processing unit 43 . The allocation processing unit 5 includes an estimated fluctuation calculation unit 50 and a divided image allocation processing unit 51.

[0023] The image scaling processing unit 30 of the overall processing unit 3 executes a scaling process to reduce the input image to an image size that can be input to the object detection process based on the deep learning result. The image scaling processing unit 30 can reduce the image size while maintaining the ratio of the width and height of the input image. For example, a bilinear interpolation method is used as a reduction method.

[0024] The object detection processing unit 31 of the overall processing unit 3 receives the input image reduced by the image scaling processing unit 30 and performs calculations on a trained object detection model based on a predetermined deep learning. The object detection processing unit 31 provides the reduced input image as an input to a prepared object detection model, performs calculations on the object detection model, and generates a set of attribute information including attribute values ​​and BB (Bounding Box) of objects included in the input image as metadata for the input image.

[0025] Attribute values ​​represent the inherent shape, form, and properties of an object. Examples of attribute values ​​include "dog," "bicycle," and "car." A bounding box (BB) has a minimum rectangular range that can circumscribe a detected object and enclose the object. The BB information included in the metadata of the input image includes at least the center coordinates (X, Y) of the BB, the height (H) of the BB, and the width (W) of the BB.

[0026] The object detection processing unit 31 detects objects included in the input image using an object detection model such as YOLO that uses a Convolutional Neural Network (CNN) that has been trained in advance by a computing device such as an external server.

[0027] The metadata scaling processing unit 32 of the overall processing unit 3 performs a scaling process to enlarge the BB included in the metadata of the input image generated by the object detection processing unit 31 so as to correspond to the size of the input image before reduction. The reason for performing this scaling process is that the metadata generated by the object detection processing unit 31 is for the reduced input image. The metadata scaling processing unit 32 scales the BB included in the metadata of the input image by using, for example, a bilinear interpolation method.

[0028] For example, if the width of the input image is W in , height H in Let the width of the input image reduced by the image scaling processor 30 be W det , height H det In this case, the metadata scaling processing unit 32 calculates the center coordinates (X bb ,Y bb ) to (X bb ×W in / W det ,Y bb ×H in / H det ) and the width of BB W bb , height H bb Each of these is W bb ×W in / W det , H bb ×H in / H det We map BB to the original input image by scaling it to

[0029] The reliability filter processing unit 33 of the overall processing unit 3 selects objects whose reliability is equal to or greater than a preset reliability threshold Th from the detected object group scaled by the metadata scaling processing unit 32. The reliability filter processing unit 33 sends metadata MD2 of objects whose reliability is equal to or greater than the reliability threshold Th to the synthesis processing unit 8. In addition, the reliability filter processing unit 33 sends metadata MD3 of objects whose reliability is less than the reliability threshold Th to the allocation processing unit 5.

[0030] The division unit 1 of the division processing unit 4 divides an input image into a plurality of divided images. The number of divisions in the width direction of the input image is N w , the number of divisions in the height direction is N h For example, in the example shown in FIG. w =2, N h =2, for a total of four split images.

[0031] The selection processing unit 2 selectively outputs some of the divided images divided by the division unit 1. The process of the selection processing unit 2 will be described in detail later.

[0032] The image scaling processing unit 40 of the segmentation processing unit 4 performs scaling processing to reduce each of the multiple segmented images output from the selection processing unit 2 to a specified image size that can be input to an object detection model based on deep learning. The image scaling processing unit 40 reduces the image size while maintaining parameter values ​​such as the ratio between the width and height of each segmented image so that it corresponds to the size of an input image of the object detection model used by the object detection processing unit 41 described below.

[0033] The object detection processing unit 41 of the segmentation processing unit 4 receives the segmented images reduced by the image scaling processing unit 40 as input, and performs calculations of a trained object detection model based on a predetermined deep learning for each segmented image. The object detection processing unit 41 provides the reduced segmented images as input to a previously prepared object detection model, performs calculations of the object detection model, and generates a set of attribute information including the attribute values ​​of the objects included in the segmented images and the BB as metadata of the segmented images. The information of the BB included in the metadata of the segmented images includes at least information on the center coordinates (X, Y) of the BB, the height (H) of the BB, and the width (W) of the BB.

[0034] The object detection processing unit 41 detects objects included in the segmented images using an object detection model such as YOLO that uses a convolutional neural network (CNN) that has been trained in advance by a computing device such as an external server.

[0035] Here, the width of the input image is W in, height H in Let the width of a specified image size that can be input to a pre-prepared object detection model be W det , height H det In this case, the number of divisions in the width direction of the input image is N w , the number of divisions in the height direction N h is given by the following equations (1) and (2). w _ max is the upper limit of the number of divisions in the width direction of the input image, N h _ max is the upper limit of the number of divisions in the height direction of the input image. N w =min(N w _ max ,ceiling(W in / W det ))···(1) N h =min(N h _ max , ceiling(H in / H det ))···(2)

[0036] The ceiling() in formula (1) and formula (2) is a function that rounds up a number to the specified number of digits. w _ max ,ceiling(W in / W det )) is N w _ max and ceiling(W in / W det ) means that the smaller value is taken.

[0037] The metadata adjustment processing unit 42 of the division processing unit 4 performs metadata adjustment processing for mapping the BB detected by the object detection processing unit 41 to the original image before division, that is, the input image, for each divided image input to the object detection processing unit 41. div Segmented images [i][j](0≦i≦N w -1,0≦j≦N h Let the center coordinate of the BB detected in (x -1) be (x bb _ div ,y bb _div ), width w bb _ div , height h bb _ div The center coordinate of the BB after adjustment to the coordinates of the original image is (x bb ,y bb ), width w bb , height h bb Let us assume that.

[0038] The metadata adjustment processing unit 42 maps the BB detected by the object detection processing unit 41 to the input image based on the following equations.

[0039] x bb =x bb _ div ×floor(W in / N w )+floor(W in / N w )×i (3) y bb =y bb _ div ×floor(H in / N h )+floor(H in / N h )×j (4) w bb =w bb _ div ×floor(W in / N w ) ···(5) h bb =h bb _ div ×floor(H in / N h ) ···(6)

[0040] In expressions (3) to (6), floor() refers to a function that rounds down a numerical value to a specified number of digits. The reliability filter processing unit 43 of the division processing unit 4 selects objects whose reliability is equal to or greater than a preset reliability threshold Th from the detected objects after the metadata has been adjusted by the metadata adjustment processing unit 42. The metadata MD1 of the objects selected by the reliability filter processing unit 43 is output to the synthesis processing unit 8 and is stored in the holding unit 6.

[0041] The selection processing unit 7 selectively outputs a portion of the detection results from among the past detection results stored in the storage unit 6. The process of the selection processing unit 7 will be described in detail later.

[0042] The synthesis processing unit 8 performs a process of interpolating objects not detected by the overall processing unit 3 with objects detected by the division processing unit 4. The synthesis processing unit 8 performs a metadata selection process of determining whether the objects detected by the overall processing unit 3 match those detected by the division processing unit 4, and determining the mismatched objects as objects to be interpolated. This metadata selection process will be described.

[0043] The synthesis processing unit 8 compares the metadata MD1 of one of the divided images with the metadata MD2 of the entire image, and judges whether or not the attribute value of the metadata MD2 of the entire image matches the attribute value of the metadata MD1 of the divided images. If they match, the synthesis processing unit 8 calculates the degree of overlap. Specifically, the synthesis processing unit 8 calculates the degree of overlap by dividing the overlapping area between the BB included in the metadata MD2 of the entire image and the BB included in the metadata MD1 of the divided images by the area of ​​the BB included in the metadata MD1 of the divided images. The synthesis processing unit 8 performs such a match / mismatch judgment and calculation of the degree of overlap for each divided image.

[0044] Then, if the degree of overlap exceeds a preset threshold, the synthesis processing unit 8 determines that the attribute information in the metadata MD1 of the divided image and the attribute information in the metadata MD2 of the entire image are the same attribute information, and removes the same attribute information from the metadata MD1 of the divided image.

[0045] Furthermore, the synthesis processor 8 performs a metadata summing process in which the metadata MD1 of the divided images after the metadata sorting process and the metadata MD2 of the entire image are combined and output as a final object detection result. In other words, the synthesis processor 8 generates the metadata MD of the input image by interpolating the metadata MD2 of the entire image with the metadata MD1 of the divided images from which overlapping attribute information has been removed.

[0046] Next, the operation of the allocation processing unit 5 and the selection processing units 2 and 7 will be described. The allocation processing unit 5 dynamically controls the frame rate at which object detection is performed for each divided image based on the object detection result of the overall processing unit 3. Specifically, the allocation processing unit 5 estimates the fluctuation in the number of detected objects for each divided image from the processing result of the overall processing unit 3, and reduces the thinning rate of frames at which object detection is performed for divided images with large estimated fluctuations (divided images with unstable object detection). In addition, the allocation processing unit 5 increases the thinning rate of frames for divided images with small estimated fluctuations (divided images with stable object detection). This avoids the difficulty of object detection due to frame thinning, and achieves a reduction in the amount of calculation while maintaining stable object detection. Even for divided images with small fluctuations in the number of detected objects, an upper limit is set for the number of frames to be thinned out so that changes due to object movement and the like can be tracked. The overall processing unit 3 executes object detection processing for the input image for each frame.

[0047] Each time object detection processing is performed on each divided image, segmentation processing unit 4 records the object detection result in storage unit 6. A synthesis processing unit 8 uses the latest object detection result for divided images for which object detection has been performed, and uses past object detection results stored in storage unit 6 for divided images for which object detection has not been performed, and performs synthesis processing with the object detection result of the entire image to obtain a final object detection result.

[0048] The estimated fluctuation is obtained from the detection result of the overall processing unit 3 in the following procedure. First, the estimated fluctuation calculation unit 50 of the assignment processing unit 5 extracts an object group whose reliability is equal to or greater than the reliability threshold Th when multiplied by α (α is a real number equal to or greater than 2) from among the object group excluded by the reliability filter processing unit 33 of the overall processing unit 3 from the object detection result of the overall processing unit 3. Specifically, the reliability filter processing unit 33 outputs metadata MD3 of objects whose reliability is less than the reliability threshold Th. The estimated fluctuation calculation unit 50 extracts an object group whose reliability is equal to or greater than the reliability threshold Th when multiplied by α from among the object group whose reliability is less than the reliability threshold Th.

[0049] Next, the estimated fluctuation calculation unit 50 checks which of the coordinate ranges of the divided images the center coordinates of each of the extracted objects are located within. Then, the estimated fluctuation calculation unit 50 increments the estimated value of the number of detected objects of the divided image including the center coordinates of the extracted object. For example, if the center coordinates of the extracted object are located within the coordinate range of divided image #0, the estimated fluctuation calculation unit 50 increments the estimated value of the number of detected objects of divided image #0. This increment of the estimated value of the number of detected objects is performed for each extracted object, and an estimated value of the number of detected objects of each divided image is obtained.

[0050] FIG. 3 is a flowchart for explaining a calculation process of an estimated value of the number of detected objects in a divided image. The estimated fluctuation calculation unit 50 sets a variable Im to 0 (step S101 in FIG. 3). The variable Im is a number assigned to each divided image and ranges from 0 to (N div The estimated fluctuation calculation unit 50 calculates the value of the variable Im as the total number of divided images N div It is determined whether the variable Im is less than the total number of divided images N div If it is not less than (No in step S102), the estimated fluctuation calculation section 50 ends the series of detected object number estimate value calculation processes.

[0051] The estimated fluctuation calculation unit 50 calculates the variable Im by dividing the total number of divided images N divIf it is less than Im (Yes in step S102), the estimated value Nobj[Im] of the number of detected objects in the divided image with number Im is set to 0 (step S103 in FIG. 3).

[0052] The estimated fluctuation calculation unit 50 sets the variable no to 0 (step S104 in FIG. 3). The variable no is a number assigned to each object detected by the overall processing unit 3 and having a reliability less than the reliability threshold Th. The estimated fluctuation calculation unit 50 determines whether the variable no is less than the extracted meta number (step S105 in FIG. 3). The extracted meta number is the number of objects detected by the overall processing unit 3 and having a reliability less than the reliability threshold Th.

[0053] If the variable no is equal to or greater than the extracted meta number (No in step S105), the estimated fluctuation calculation unit 50 increments the variable Im by 1 (step S106 in FIG. 3). After the process of step S106 ends, the estimated fluctuation calculation unit 50 returns to the process of step S102.

[0054] On the other hand, if the variable no is less than the extracted meta number (Yes in step S105), the estimated fluctuation calculation unit 50 judges whether the central coordinates of the object with the number no among the group of objects whose reliability is less than the reliability threshold Th are within the coordinate range of the divided image with the number Im (step S107 in FIG. 3). Note that the coordinates on the boundary line of each divided image are the coordinates of only one divided image, and do not overlap.

[0055] When the central coordinates of the object numbered no are within the coordinate range of the divided image numbered Im (Yes in step S107), the estimated fluctuation calculation unit 50 calculates the expected reliability value by multiplying the reliability of the object numbered no by a predetermined coefficient α (step S108 in FIG. 3). Here, the coefficient α is a value that represents the rate at which the reliability is improved by reducing and mitigating the reduction of characteristic parts of the object by the divided process, compared to the whole process.

[0056] The estimated fluctuation calculation unit 50 judges whether the expected reliability value calculated in step S108 is equal to or greater than the reliability threshold Th (step S109 in FIG. 3). If the expected reliability value is equal to or greater than the reliability threshold Th (Yes in step S109), the estimated fluctuation calculation unit 50 increments the estimated value Nobj[Im] of the number of detected objects in the divided image numbered Im by one (step S110 in FIG. 3).

[0057] Next, the estimated fluctuation calculation unit 50 increments the variable no by one (step S111 in FIG. 3) and returns to step S105. If the central coordinate of the object with the number no is not within the coordinate range of the divided image with the number Im (No in step S107) or if the expected reliability value is less than the reliability threshold Th (No in step S109), the estimated fluctuation calculation unit 50 skips the process to step S111.

[0058] In this way, the estimated fluctuation calculation section 50 can obtain the estimated value Nobj of the number of detected objects in each divided image by executing the series of processes shown in FIG.

[0059] Next, the estimated fluctuation calculation unit 50 obtains an estimated fluctuation value for each divided image by calculating, for each divided image, the absolute difference between the estimated value Nobj of the number of detected objects in that divided image and the estimated value Nobj of the number of detected objects in that divided image in the immediately preceding frame.

[0060] Here, the coefficient α is the ratio of the reliability of the detection of the same object in the whole image to the reliability of the detection in the divided images. In this embodiment, α is a single real value determined in advance, but it may be a different value for each coordinate range of the divided images. In addition, α may be corrected based on the error between the estimated value of the number of detected objects in the divided images and the actual number of detected objects.

[0061] The frame thinning rate is controlled by assigning frames and computing cores to perform object detection for each divided image in a period of N frames (N is the total number of divided images). divHere, the calculation core is a calculation core that functions as the object detection processing unit 41, and in order to reduce the amount of calculation, the number of calculation cores is determined to be equal to or larger than the total number of divided images N. div That is, the object detection processing units 41, the image scaling processing units 40, the metadata adjustment processing units 42, and the reliability filter processing units 43 are provided in numbers equal to the number of the processing cores.

[0062] FIG. 4 is a diagram showing an example of the allocation process. Here, the total number of divided images N div = 4, N = 4, and the number of calculation cores functioning as the object detection processing unit 41 is 2. The vertical axis of FIG. 4 indicates the frame thinning rate Fr, and the horizontal axis indicates the frame number. In the example of FIG. 4, the upper limit Fr max is set to 0.75. In the initial state, the frame thinning rate Fr is uniform among the divided images.

[0063] First, the estimated fluctuation calculation unit 50 calculates an estimated value of the fluctuation in the number of detected objects in each divided image for each frame in one period (for example, frames numbered #0 to #3) from the processing result of the overall processing unit 3, and calculates a cumulative value of this estimated value of the fluctuation for one period for each divided image. The method of calculating the estimated value of the fluctuation in the number of detected objects in each divided image is as described above.

[0064] Then, when the estimated fluctuation calculation unit 50 reaches the last frame in one period (for example, frame number #3), it divides the cumulative value of the estimated value of the fluctuation for each divided image by N (=4) to obtain the average value E ave [Im] is calculated for each divided image.

[0065] The divided image allocation processing unit 51 of the allocation processing unit 5 calculates the average value E of the estimated fluctuation of the number of detected objects for each divided image calculated by the estimated fluctuation calculation unit 50. ave Based on [Im], the selection processing unit 2 reserves the computation cores and divided images for each frame that will be the target of object detection in the next cycle so that the number of times (allocation amount) that object detection is executed is proportionally distributed to each divided image.

[0066] As mentioned above, the average estimated value of the fluctuation of the number of detected objects E ave For divided images with a small [Im], the frame thinning rate Fr is high. On the other hand, the average estimated value of the fluctuation of the number of detected objects E ave A divided image with a large [Im] has a low frame thinning rate Fr. In this way, dynamic control of the frame thinning rate Fr for each divided image is achieved.

[0067] 5 is a flowchart for explaining the operation of the divided image allocation processing unit 51. First, the divided image allocation processing unit 51 calculates the allocatable amount J n (Step S200 in FIG. 5). total is an integer value obtained by multiplying N by the number of computing cores, and is the number of object detections that can be performed in one cycle. In the above example, N=4 and the number of computing cores is 2, so the total allocatable capacity J total At the time of step S200, the allocable amount J n is the total amount that can be allocated, J total is set equal to

[0068] The divided image allocation processing unit 51 sets the allocation amount J[Im] of each divided image to 0 (Step S201 in FIG. 5). The allocation amount J[Im] is a value that determines for each divided image the number of times object detection is performed for one divided image within one period.

[0069] Next, the divided image allocation processing unit 51 calculates the allocation amount J[Im] of all divided images by the upper limit Fr max The minimum allocation J corresponds to min (Step S202 in FIG. 5). As a result, the frame thinning rate Fr of all divided images is set to the upper limit Fr max In this embodiment, the minimum allocation amount J min Let M be the minimum allocation amount J. M is a predetermined integer of 1 or more. min By setting M, all the divided images are selected as the target for object detection at least M times during one period.

[0070] The divided image allocation processing unit 51 sets the allocation amount J[Im] of each divided image to the minimum allocation amount J min Therefore, the current available quota is J n From N div xJ min The latest allocation amount J is calculated by subtracting the multiplication result of n (Step S203 in FIG. 5).

[0071] Next, the divided image allocation processing unit 51 calculates an average value E ave It is checked whether [Im] is 0 or not (step S204 in FIG. 5). The average value E ave When [Im] is 0 for all divided images, it means that there is no fluctuation in the number of detected objects in all divided images. n (the number of times object detection can be performed) is evenly distributed to the allocation J[Im] of each divided image (Step S205 in FIG. 5).

[0072] In the equal distribution in step S205, the divided image allocation processing unit 51 calculates the allocation amount J[Im] of each divided image as follows. J new [Im]=J old [Im]+Floor(J n / N div ) · · · (7)

[0073] J old [Im] is the allocation of each divided image before distribution, J new [Im] is the allocation amount of each divided image after distribution. In addition, the divided image allocation processing unit 51 calculates the average value E ave If [Im] is not 0 (No in step S204), the remaining allocable amount J n Let E be the average of the estimated fluctuations. ave The image allocation amount J[Im] is weighted based on the ratio of [Im] and distributed to each divided image (step S206 in FIG. 5).

[0074] In the proportional distribution in step S206, the divided image allocation processing unit 51 calculates the allocation amount J[Im] of each divided image as follows. J new [Im]=J old [Im] +min(N,Floor(J n ×(E ave [Im] / ΣE ave [Im])) (8)

[0075] ΣE in Equation (8) ave Needless to say, [Im] is the sum of the average estimated fluctuations in the number of detected objects in all the divided images. n ×(E ave [Im] / ΣE ave [Im])) is N and Floor(J n ×(E ave [Im] / ΣE ave [Im]) or the smaller value.

[0076] In addition, the unallocated amount that could not be distributed in step S205 or S206 (the remaining allocable amount J n (a value obtained by subtracting the sum of the allocation amounts J[Im] of the divided images from N) may occur. In this case, the divided image allocation processing unit 51 may distribute the unallocated amount equally to all divided images within a range not exceeding N, or may distribute the unallocated amount to divided images with larger or smaller allocation amounts J[Im].

[0077] When the allocation capacity J[Im] of each divided image is determined by step S205 or S206, the divided image allocation processing unit 51 reserves the calculation core that will perform the object detection processing and the divided images and frames that will be the subject of the object detection processing to the selection processing units 2 and 7, in order of the divided images with the largest allocation capacity J[Im] (step S207 in Figure 5).

[0078] 6 is a flowchart for explaining the reservation process in step S207. First, the divided image allocation processing unit 51 sets the reservation completion flag of each divided image to 0 (step S300 in FIG. 6). Next, the divided image allocation processing unit 51 determines whether the reservation completion flag of each divided image is 1 (step S301 in FIG. 6).

[0079] In this case, the reservation completion flag of each divided image is 0, so the process proceeds to step S302. The divided image allocation processing unit 51 searches for a divided image with the maximum allocation amount J[Im] among the divided images with the reservation completion flag set to 0 as the reservation target (step S302 in FIG. 6). The number of the searched divided image to be reserved is set to i, and the allocation amount is set to J[i].

[0080] Then, the divided image allocation processing unit 51 allocates N d J[i] / N for each frame d In order to execute the object detection process only once, an unreserved processing core is reserved for the object detection process of the reserved divided image (step S303 in FIG. 6). d is an integer equal to or greater than 1 and less than N. However, in this reservation, the same divided image can only be reserved once per frame.

[0081] The divided image allocation processing unit 51 sets the reservation completion flag of the divided image with the number i for which reservation has been completed to 1 (step S304 in FIG. 6), and returns to step S301. In this way, divided images whose reservation completion flag is 0 (reservation not yet completed) are reserved in descending order of the allocation amount J[Im], and processing cores are allocated to them.

[0082] When the reservation completion flags of all the divided images become 1 (Yes in step S301), the reservation process (step S207) ends. In addition, the divided image allocation processing unit 51 selects N d When reserving for each frame, this N d If there are no unreserved computing cores within the frame period, the next N dAny unreserved computing core within the frame period may be reserved for object detection processing of the segmented image to be reserved.

[0083] A specific example of the reservation process will be described with reference to Figs. 4 and 7. The average value E ave From the calculation result of [Im], when reserving the processing cores for the next cycle (frames #4 to #7) for object detection processing of the divided images, the allocation J[0] of the divided image #0 is 1, the allocation J[1] of the divided image #1 is 4, the allocation J[2] of the divided image #2 is 2, and the allocation J[3] of the divided image #3 is 1. d Let's say it's 2.

[0084] First, for the divided image #1 with the largest allocation amount J[1], the divided image allocation processing unit 51 reserves the processing core #0 twice for frames #4 to #5, and further reserves the processing core #0 twice for frames #6 to #7. Next, for the divided image #2 with the second largest allocation amount J[2], the divided image allocation processing unit 51 reserves the processing core #1 once for frames #4 to #5, and further reserves the processing core #1 once for frames #6 to #7.

[0085] Next, the divided image allocation processing unit 51 reserves the divided image with the smaller number from among the divided images with numbers #0 and #3 having the same allocation amounts J[0] and J[3]. Specifically, the divided image allocation processing unit 51 reserves the processing core with number #1 once for the divided image with number #0 in frames with numbers #4 to #5. Finally, the divided image allocation processing unit 51 attempts to reserve the processing core with number #4 to #5 for the divided image with number #3, but since there are no unreserved processing cores in frames with numbers #4 to #5, it reserves the processing core with number #1 once for frames with numbers #6 to #7. FIG. 7 shows the reservation results described above.

[0086] Next, the operations of the selection processors 2 and 7, the division processor 4, the storage unit 6, and the synthesis processor 8 in response to reservation processing will be described.

[0087] In the next period, a divided image is selected for each frame and object detection processing is performed based on the reservation processing by the divided image allocation processing unit 51. For example, based on the reservation result for the frame numbered #4 in Fig. 7, the selection processing unit 2 outputs the divided image numbered #1 to the image scaling processing unit 40 corresponding to the calculation core numbered #0 (object detection processing unit 41 numbered #0) so that object detection processing is performed on the divided image numbered #1 by the calculation core numbered #0 (object detection processing unit 41 numbered #0) and the image scaling processing unit 40, metadata adjustment processing unit 42, and reliability filter processing unit 43 corresponding to this calculation core.

[0088] In addition, based on the reservation result for frame number #4, the selection processing unit 2 outputs the divided image numbered #2 to the image scaling processing unit 40 corresponding to the calculation core numbered #1 so that object detection processing is performed on the divided image numbered #2 by the calculation core numbered #1 (object detection processing unit 41 numbered #1) and the image scaling processing unit 40, metadata adjustment processing unit 42, and reliability filter processing unit 43 corresponding to this calculation core.

[0089] Metadata MD1, which is the object detection result for the divided images numbered #1 and #2, is output to the synthesis processing unit 8 and the storage unit 6. The storage unit 6 stores the metadata MD1.

[0090] Based on the reservation result for frame number #4, for split images numbered #0 and #3 for which object detection processing is not performed on frame number #4, the selection processing unit 7 outputs the most recent detection results among the past object detection results (metadata MD1) for split images numbered #0 and #3 stored in the holding unit 6 to the synthesis processing unit 8 as the detection result for frame number #4.

[0091] The synthesis processing unit 8 performs the above metadata selection process on the metadata MD1 of the divided images numbered #1 and #2 output from the reliability filter processing unit 43 and the metadata MD1 of the divided images numbered #0 and #3 output from the selection processing unit 7. The synthesis processing unit 8 then combines the metadata MD1 after the metadata selection process with the object detection result (metadata MD2) of the overall processing unit 3 for the frame numbered #4, and outputs the metadata MD that is the final object detection result.

[0092] Next, based on the reservation result for frame numbered #5 in Figure 7, the selection processing unit 2 outputs the divided image numbered #1 to the image scaling processing unit 40 corresponding to the calculation core numbered #0 (object detection processing unit 41 numbered #0) so that object detection processing is performed on the divided image numbered #1 by the calculation core numbered #0 (object detection processing unit 41 numbered #0) and the image scaling processing unit 40, metadata adjustment processing unit 42 and reliability filter processing unit 43 corresponding to this calculation core.

[0093] In addition, based on the reservation result for frame numbered #5, the selection processing unit 2 outputs the divided image numbered #0 to the image scaling processing unit 40 corresponding to the calculation core numbered #1 so that object detection processing is performed on the divided image numbered #0 by the calculation core numbered #1 (object detection processing unit 41 numbered #1) and the image scaling processing unit 40, metadata adjustment processing unit 42, and reliability filter processing unit 43 corresponding to this calculation core.

[0094] Metadata MD1, which is the object detection result for the divided images numbered #1 and #0, is output to the synthesis processing unit 8 and the storage unit 6. Based on the reservation result for frame number #5, for split images numbered #2 and #3 for which object detection processing is not performed on frame number #5, the selection processing unit 7 outputs the most recent detection results among the past object detection results (metadata MD1) for split images numbered #2 and #3 stored in the holding unit 6 to the synthesis processing unit 8 as the detection results for frame number #5.

[0095] The synthesis processing unit 8 performs the above metadata selection process on the metadata MD1 of the divided images numbered #1 and #0 output from the confidence filter processing unit 43 and the metadata MD1 of the divided images numbered #2 and #3 output from the selection processing unit 7. The synthesis processing unit 8 then combines the metadata MD1 after the metadata selection process with the object detection result (metadata MD2) of the overall processing unit 3 for the frame numbered #5, and outputs the metadata MD that is the final object detection result. The same process may be carried out for the subsequent frames.

[0096] Next, the process flow of the object detection device of this embodiment will be described with reference to the flowchart of Fig. 8. First, an input image is input to the object detection device (step S400 in Fig. 8). For example, an image captured by an external camera (not shown) is input to the object detection device.

[0097] The overall processing unit 3 executes overall processing for performing object detection on the entire input image (Step S401 in FIG. 8).

[0098] The allocation processing unit 5 executes an allocation process of determining in advance, at a period of N frames, divided images for each frame that are to be subject to object detection among the plurality of divided images, and reserving the determined divided images (Step S402 in FIG. 8).

[0099] The division unit 1 of the division processing unit 4 divides the input image into a plurality of divided images (Step S403 in FIG. 8). Based on the decision of the allocation processing unit 5, the selection processing unit 2 outputs the divided images for which the object detection process is to be performed, out of all the divided images of the frame to be processed, to the image scaling processing unit 40 (Step S404 in FIG. 8).

[0100] The image scaling processing unit 40, object detection processing unit 41, metadata adjustment processing unit 42, and reliability filter processing unit 43 of the segmentation processing unit 4 execute object detection processing on the segmented images output from the selection processing unit 2 (step S405 in FIG. 8).

[0101] As is clear from the above description, the allocation processing unit 5 determines the allocation for the next cycle. Therefore, the selection processing unit 2, the image scaling processing unit 40, the object detection processing unit 41, the metadata adjustment processing unit 42, and the reliability filter processing unit 43 execute the object detection process on the divided images based on the processing result of the overall processing unit 3 in the previous cycle.

[0102] The selection processing unit 7 selects past object detection results for divided images for which object detection processing is not performed from among the object detection results stored in the storage unit 6, and outputs the results to the synthesis processing unit 8 (Step S406 in FIG. 8).

[0103] The synthesis processing unit 8 performs synthesis processing to combine the object detection result output from the overall processing unit 3, the object detection result output from the division processing unit 4, and the past object detection result output from the selection processing unit 7 (step S407 in FIG. 8). Then, the synthesis processing unit 8 outputs metadata MD, which is the final object detection result (step S408 in FIG. 8). The object detection device performs the above process in FIG. 8 for each frame of the input image.

[0104] 9 is a flow chart for explaining details of the overall process in step S401. The image scaling processing unit 30 of the overall processing unit 3 executes scaling processing on the input image (step S500 in FIG. 9).

[0105] The object detection processing unit 31 of the overall processing unit 3 executes object detection processing on the entire input image reduced by the image scaling processing unit 30 (step S501 in FIG. 9).

[0106] The metadata scaling processing unit 32 of the overall processing unit 3 performs a scaling process to enlarge the BB included in the metadata of the input image generated by the object detection processing unit 31 so that it corresponds to the size of the input image before reduction (step S502 in FIG. 9).

[0107] The reliability filter processing unit 33 of the overall processing unit 3 sends the detection results (metadata MD2) of objects whose reliability is equal to or greater than the reliability threshold Th to the synthesis processing unit 8, and sends the detection results (metadata MD3) of objects whose reliability is less than the reliability threshold Th to the allocation processing unit 5 (Figure 9 step S503).

[0108] Fig. 10 is a flow chart for explaining the details of the allocation process in step S402. The estimated fluctuation calculation unit 50 of the allocation processing unit 5 calculates the estimated value Nobj of the number of detected objects in each divided image for each frame based on the detection result of the object whose reliability is less than the reliability threshold Th (Fig. 10, step S600). The details of this process are as explained in Fig. 3.

[0109] The estimated fluctuation calculation unit 50 calculates an estimated value of the fluctuation between frames in the number of detected objects for each divided image by calculating, for each divided image, the absolute difference between the estimated value Nobj of the number of detected objects for each divided image and the estimated value Nobj of the number of detected objects for each divided image in the immediately preceding frame (FIG. 10, step S601).

[0110] Then, the estimated fluctuation calculation unit 50 calculates the average value E ave [Im] is calculated for each divided image (step S602 in FIG. 10).

[0111] The divided image allocation processing unit 51 of the allocation processing unit 5 calculates the average value E of the estimated fluctuation of the number of detected objects in each divided image calculated by the estimated fluctuation calculation unit 50. ave Based on [Im], the allocation amount J[Im] of each divided image is calculated (step S603 in FIG. 10).

[0112] The divided image allocation processing unit 51 reserves the computation cores and frames for executing the object detection process for the selection processing units 2 and 7 in the order of the divided images with the largest allocation amount J[Im] (step S207 in FIG. 5). The details of the process of the divided image allocation processing unit 51 are as described in FIG. 5 and FIG. 6.

[0113] FIG. 11 is a flowchart illustrating the details of the object detection process for the divided images in step S405. The image scaling processing unit 40 of the division processing unit 4 executes a scaling process for reducing each of the plurality of divided images output from the selection processing unit 2 to a specified image size (Step S700 in FIG. 11).

[0114] The object detection processing unit 41 of the segmentation processing unit 4 executes object detection processing on the segmented images reduced by the image scaling processing unit 40 (Step S701 in FIG. 11).

[0115] The metadata adjustment processing unit 42 of the segmentation processing unit 4 performs metadata adjustment processing for each segmented image input to the object detection processing unit 41 to map the BB detected by the object detection processing unit 41 to the input image before segmentation (Figure 11 step S702).

[0116] The reliability filter processing unit 43 of the splitting processing unit 4 outputs the detection results (metadata MD1) of objects whose reliability is equal to or greater than the reliability threshold Th from among the detection results after the metadata has been adjusted by the metadata adjustment processing unit 42 to the synthesis processing unit 8 and the storage unit 6 (Figure 11 step S703).

[0117] The holding unit 6 stores the metadata MD1 output from the reliability filter processing unit 43 (Step S704 in FIG. 11).

[0118] Fig. 12 is a flowchart for explaining the details of the synthesis process in steps S407 and 408. The synthesis process unit 8 executes a metadata selection process on the object detection result (metadata MD1) output from the division process unit 4 and the object detection result (metadata MD1) output from the selection process unit 7 (Fig. 12, step S800).

[0119] The flow of the metadata selection process in step S800 will now be described in more detail with reference to the flowchart in FIG. First, the object detection result (metadata MD2) output from the overall processing unit 3 is input to the synthesis processing unit 8 (step S900 in FIG. 13). In addition, the object detection result (metadata MD1) output from the division processing unit 4 and the object detection result (metadata MD1) output from the selection processing unit 7 are input to the synthesis processing unit 8 (step S901 in FIG. 13).

[0120] Next, the synthesis processor 8 judges whether or not the attribute value of the metadata MD2 of the entire image matches the attribute value of the metadata MD1 of the divided images (step S902 in FIG. 13). The synthesis processor 8 calculates the overlapping degree between the metadata MD2 and the metadata MD1 whose attribute value matches that of the metadata MD2 (step S903 in FIG. 13). Specifically, the synthesis processor 8 determines the overlapping degree as a value obtained by dividing the overlapping area between the BB included in the metadata MD2 and the BB included in the metadata MD1 by the area of ​​the BB included in the metadata MD1.

[0121] In addition, in step S902, if the attribute value included in the metadata MD2 of the entire image does not match the attribute value included in the metadata MD1 of the divided image (No in step S902), processing is performed on other attribute information included in the metadata MD1 of the same divided image.

[0122] Next, when the overlap calculated in step S903 exceeds a preset threshold value (Yes in step S904 in FIG. 13), the synthesis processor 8 determines that the attribute information in the metadata MD1 of the divided images and the attribute information in the metadata MD2 of the entire image are identical, and removes the identical attribute information from the metadata MD1 (step S905 in FIG. 13). That is, the attribute information of the metadata in which the attribute values ​​of the objects detected in the divided images and the entire image are the same and the objects are in a corresponding positional relationship in the input image is removed from the metadata MD1 of the divided images.

[0123] On the other hand, if the degree of overlap is equal to or less than the threshold (No in step S904), the process ends. In this manner, the synthesis processing unit 8 executes the metadata selection process of FIG.

[0124] Next, the synthesis processing unit 8 synthesizes the metadata MD1 after the metadata selection processing and the metadata MD2 of the entire image, and outputs the metadata MD that is the final object detection result (Step S801 in FIG. 12).

[0125] As described above, in this embodiment, it is possible to reduce the amount of calculation while suppressing the occurrence of missed object detection.

[0126] In this embodiment, the detection result of a divided image that is not selected in a certain frame as a result of the allocation is treated as a past detection result stored in the storage unit 6. In contrast to this, as another example, a motion vector value of an object in each divided image may be calculated for each frame, and the past detection result for the above-mentioned non-selected divided image may be corrected with the calculated motion vector value (the central coordinates of BB in the past detection result may be corrected with the motion vector value).

[0127] [Second Example] Next, a second embodiment of the present invention will be described. Fig. 14 is a block diagram showing the configuration of an object detection device according to the second embodiment of the present invention. The object detection device of this embodiment includes a division unit 1, selection processing units 2 and 7, an overall processing unit 3, a division processing unit 4, an allocation processing unit 5a, a storage unit 6, and a synthesis processing unit 8.

[0128] The division unit 1, the selection processing units 2 and 7, the overall processing unit 3, the division processing unit 4, the holding unit 6, and the synthesis processing unit 8 are as described in the first embodiment. The allocation processing unit 5 a of this embodiment includes an estimated fluctuation calculation unit 50 , a divided image allocation processing unit 51 a , and a scene change detection processing unit 52 .

[0129] The object detection device of this embodiment performs object detection processing in the same manner as in the first embodiment, but differs from the first embodiment in that when an input image changes suddenly within a period, reserved computing cores and frames are cancelled and returned to their initial states. For this processing, in this embodiment, a scene change detection processing unit 52 is provided in the allocation processing unit 5a.

[0130] Fig. 15 is a flowchart explaining the operation of the scene change detection processing unit 52. The scene change detection processing unit 52 searches for an object having the same attribute value as an object included in the latest detection result output from the segmentation processing unit 4 from among the past detection results stored in the storage unit 6 (Fig. 15, step S1000).

[0131] Then, the scene change detection processing unit 52 calculates the IoU (Intersection over Union) of objects with the same attribute value included in the latest detection result and the past detection results (FIG. 15, step S1001). The IoU is a value obtained by dividing the common area of ​​the BB area of ​​the object included in the latest detection result and the BB area of ​​the object included in the past detection result by the union of the BB area of ​​the object included in the latest detection result and the BB area of ​​the object included in the past detection result. The scene change detection processing unit 52 calculates such an IoU for each object with the same attribute value.

[0132] The scene change detection processing unit 52 determines whether a state in which the average value of IoU calculated for each object is less than a predetermined threshold value β has occurred for γ consecutive frames (γ is a predetermined integer equal to or greater than 2) (FIG. 15, step S1002). The scene change detection processing unit 52 performs such a determination for each frame.

[0133] If a state in which the average IoU value is less than the threshold value β occurs for γ consecutive frames (Yes in step S1002), the scene change detection processing unit 52 determines that a sudden change has occurred in the input image, and notifies the divided image allocation processing unit 51a of the determination result (FIG. 15, step S1003).

[0134] The divided image allocation processing unit 51a, which has received the notification from the scene change detection processing unit 52, cancels all reservations of arithmetic cores and frames made to the selection processing units 2 and 7, and returns to the initial state (no reservations) (step S1004 in FIG. 15). Then, the divided image allocation processing unit 51a resets the frame in which it has been determined that a sudden change has occurred in the input image to the starting point of the cycle (for example, frames numbered #0, #4, and #8 in the example in FIG. 4), and resumes the processing in FIG. 10 (step S1005 in FIG. 15). The other configurations are as described in the first embodiment.

[0135] The object detection device described in the first and second embodiments can be realized by a computer equipped with a CPU (Central Processing Unit), a storage device, and an interface, and a program that controls these hardware resources. An example of the configuration of this computer is shown in Fig. 16. The computer is equipped with a CPU 200, a storage device 201, and an interface device (I / F) 202. The CPU 200 is equipped with multiple arithmetic cores.

[0136] For example, a camera or the like is connected to the I / F 202. In such a computer, a program for realizing the object detection method of the present invention is stored in the storage device 201. The CPU 200 executes the processes described in the first and second embodiments according to the program stored in the storage device 201. [Industrial Applicability]

[0137] The present invention can be applied to image processing techniques. [Explanation of symbols]

[0138] 1...division unit, 2...selection processing unit, 3...overall processing unit, 4...division processing unit, 5, 5a...allocation processing unit, 6...storage unit, 7...selection processing unit, 8...synthesis processing unit, 30...image scaling processing unit, 31...object detection processing unit, 32...metadata scaling processing unit, 33...reliability filter processing unit, 40...image scaling processing unit, 41...object detection processing unit, 42...metadata adjustment processing unit, 43...reliability filter processing unit, 50...estimated fluctuation calculation unit, 51, 51a...division image allocation processing unit, 52...scene change detection processing unit.

Claims

1. a segmentation unit configured to segment an input image and output a plurality of segmented images; an overall processing unit configured to perform an object detection process on a reduced image obtained by scaling the entire input image for each frame; an allocation processing unit configured to determine in advance, among the plurality of divided images, divided images for each frame that are to be subject to object detection at a period of N frames (N is an integer equal to or greater than the total number of divided images) and reserve the determined divided images; a first selection processing unit configured to select and output, for each frame, a portion of the divided images that are to be subject to object detection from among the plurality of divided images based on a result of the reservation made by the allocation processing unit; A segmentation processing unit configured to perform an object detection process on each of the segmented images output from the first selection processing unit; A storage unit configured to store an object detection result by the segmentation processing unit; a second selection processing unit configured to select and output, for each frame, a past object detection result stored in the storage unit for a divided image for which object detection is not performed based on a result of the reservation made by the allocation processing unit; an object detection device comprising: a synthesis processing unit configured to obtain a final object detection result by synthesizing, on a frame-by-frame basis, the object detection result by the overall processing unit, the object detection result by the segmentation processing unit, and the object detection result output from the second selection processing unit.

2. 2. The object detection device according to claim 1, The allocation processing unit: an estimated fluctuation calculation unit configured to calculate an estimate of the number of detected objects for each of the divided images for each frame based on the object detection result by the overall processing unit, calculate an estimate of fluctuation between frames of the number of detected objects for each of the divided images based on an absolute difference between the estimates between frames, and calculate an average value of the estimate of fluctuation of the number of detected objects within the period for each of the divided images; an object detection device comprising: a divided image allocation processing unit configured to reserve, to the first selection processing unit, divided images for each frame that will be subject to object detection in the next period, so that the number of times object detection is performed is proportionally distributed to each divided image based on the average value of the estimated value of the fluctuation in the number of detected objects for each divided image.

3. 3. The object detection device according to claim 2, The object detection device is characterized in that the divided image allocation processing unit determines that all of the divided images will be selected as targets for object detection at least M (M is an integer greater than or equal to 1) times during the next period, and then determines the divided images for each frame that will be the targets for object detection in the next period for the remaining allocable frames, based on the average value of the estimated fluctuation in the number of detected objects for each divided image, so that the number of times object detection is performed is distributed proportionally to each divided image.

4. 4. The object detection device according to claim 2, The estimated fluctuation calculation unit calculates an estimate of the number of detected objects for each divided image by tallying up, for each divided image in which the objects are located, the number of detected objects among the group of objects detected by the overall processing unit, whose reliability calculated by the overall processing unit is less than a threshold and greater than or equal to a threshold / α (α is a real number greater than or equal to 2).

5. The object detection device according to any one of claims 1 to 4, a scene change detection processing unit configured to determine whether the input image has changed based on a latest object detection result and a past object detection result by the segmentation processing unit, The object detection device, wherein the allocation processing unit cancels the reservation and returns to an initial state when it is determined that the input image has changed.

6. A first step of dividing an input image to output a plurality of divided images; a second step of performing an object detection process on a reduced image obtained by scaling the entire input image for each frame; a third step of determining in advance, from among the plurality of divided images, divided images for each frame that are to be subjected to object detection at a period of N frames (N is an integer equal to or greater than the total number of divided images) and reserving the determined divided images; a fourth step of selecting and outputting, for each frame, a portion of the plurality of divided images that are to be subject to object detection, based on a result of the reservation made in the third step; a fifth step of performing an object detection process on each of the divided images output in the fourth step; a sixth step of storing the object detection result obtained by the fifth step; a seventh step configured to select and output the past object detection results stored in the sixth step for each frame for a divided image for which object detection is not performed based on a result of the reservation made in the third step; and an eighth step of obtaining a final object detection result by combining, for each frame, the object detection result from the second step, the object detection result from the fifth step, and the object detection result output by the seventh step.

Images